Some or all of the information on this page might not apply to Trusted Cloud by S3NS. See Differences from Google Cloud for more details.

PromQL for Cloud Monitoring

This document describes how to configure Grafana to read metric data from Cloud Monitoring. You can then use PromQL to visualize and chart data. Grafana and Cloud Monitoring are linked together by a process called a data source syncer, which enables the communication between a Grafana data source and Cloud Monitoring. The data source syncer does the following:

Sends configuration values to a Grafana data source.
Maintains authentication credentials for a Trusted Cloud by S3NS service account that can read metric data from Cloud Monitoring.

This document describes how to set up the data source syncer and provides information specific to using PromQL in Cloud Monitoring. This document assumes that you are already familiar with Grafana.

Authorize Grafana to read metric data

This section describes how to set up authorization between Cloud Monitoring and Grafana by using a data source syncer to generate and sync credentials.

Use the data source syncer for authorization

Trusted Cloud by S3NS APIs all require authentication using OAuth2; however, Grafana doesn't support OAuth2 authentication for service accounts used with Prometheus data sources. To use Grafana with Cloud Monitoring, you use the data source syncer to generate OAuth2 credentials for your service account and sync them to Grafana through the Grafana data source API.

You must use the data source syncer to configure and authorize Grafana to query data globally. If you don't follow these steps, then Grafana only executes queries against data in the local Prometheus server.

The data source syncer is a command-line interface tool that remotely sends configuration values to a given Grafana Prometheus data source. This ensures that your Grafana data source has the following configured correctly:

Authentication, done by refreshing an OAuth2 access token periodically
The Cloud Monitoring API set as the Prometheus server URL
The HTTP method set to GET
The Prometheus type and version set to a minimum of 2.40.x
The HTTP and Query timeout values set to 2 minutes

The data source syncer must run repeatedly. As Trusted Cloud by S3NS API access tokens have a lifetime of one hour, running the data source syncer every 10 minutes ensures you have an uninterrupted authenticated connection between Grafana and the Cloud Monitoring API.

Configure and authenticate the Grafana data source

You can use Grafana to query Cloud Monitoring metric data from Google Kubernetes Engine services or other environments. The instructions use editable variables to create runnable commands. We strongly recommend using the editable variables and clickable copy-paste icons embedded in the code samples.

GKE

To deploy and run the data source syncer in a Kubernetes cluster, do the following:

Choose a project, cluster, and namespace to deploy the data source syncer in.
Next, make sure you properly configure and authorize the data source syncer:
- If you're using Workload Identity Federation for GKE, then follow the instructions to create and authorize a service account. Make sure to bind it to the Kubernetes namespace in which you want to run the data source syncer.
- If you're not using Workload Identity Federation for GKE, then verify you have not modified the default Compute Engine service account.
Determine the URL of your Grafana instance, for example https://yourcompanyname.grafana.net for a Grafana Cloud deployment or http://grafana.NAMESPACE_NAME.svc:3000 for a local instance configured using the test deployment YAML.

If you deploy Grafana locally and your cluster is configured to secure all in-cluster traffic by using TLS, you need to use https:// in your URL and authenticate using one of the supported TLS authentication options.
Choose the Grafana Prometheus data source that you would like to use for Cloud Monitoring, which can be either a new or a pre-existing data source, and then find and write down the data source UID. The data source UID can be found in the last part of the URL when exploring or configuring a data source, for example https://yourcompanyname.grafana.net/connections/datasources/edit/GRAFANA_DATASOURCE_UID. Do not copy the entire datasource URL. Copy only the unique identifier in the URL.
Set up a Grafana service account by creating the service account and generating a token for the account to use:
1. In the Grafana navigation sidebar, click Administration > Users and Access > Service Accounts.
2. Create the service account by clicking Add service account, giving it a name, and granting it the "Admin" role in Grafana. If your version of Grafana allows more granular permissions, then you can use the Data Sources > Writer role.
3. Click Add service account token.
4. Set the token expiration set to "No expiration" and click Generate token, then copy the generated token to the clipboard for use as GRAFANA_SERVICE_ACCOUNT_TOKEN in the next step.

Set up the following environment variables using the results of the previous steps:

# These values are required.
PROJECT_ID=SCOPING_PROJECT_ID # The value from Step 1.
GRAFANA_API_ENDPOINT=GRAFANA_INSTANCE_URL # The value from step 2. This is a URL.
DATASOURCE_UIDS=GRAFANA_DATASOURCE_UID # The value from step 3. This is not a URL.
GRAFANA_API_TOKEN=GRAFANA_SERVICE_ACCOUNT_TOKEN # The value from step 4.
GCM_ENDPOINT_OVERRIDE=--gcm-endpoint-override="https://monitoring.s3nsapis.fr/v1/projects/PROJECT_ID/location/global/prometheus/"

Run the following command to create a CronJob that refreshes the data source on initialization and then every 10 minutes. If you're using Workload Identity Federation for GKE, then the value of NAMESPACE_NAME should be the same namespace that you previously bound to the service account.

curl https://raw.githubusercontent.com/GoogleCloudPlatform/prometheus-engine/v0.15.3/cmd/datasource-syncer/datasource-syncer.yaml \
| sed 's|$DATASOURCE_UIDS|'"$DATASOURCE_UIDS"'|; s|$GRAFANA_API_ENDPOINT|'"$GRAFANA_API_ENDPOINT"'|; s|$GRAFANA_API_TOKEN|'"$GRAFANA_API_TOKEN"'|; s|$PROJECT_ID|'"$PROJECT_ID"'|;' s|$GCM_ENDPOINT_OVERRIDE|'"$GCM_ENDPOINT_OVERRIDE"'|; \
| kubectl -n NAMESPACE_NAME apply -f -

Go to your newly configured Grafana data source and verify the Prometheus server URL value starts with https://monitoring.s3nsapis.fr. You might have to refresh the page. Once verified, go to the bottom of the page and select Save & test. You need to select this button at least once to ensure that label autocompletion in Grafana works.

Verify service account credentials

If your Kubernetes cluster has Workload Identity Federation for GKE enabled, then you can skip this section.

When running on GKE, Cloud Monitoring automatically retrieves credentials from the environment based on the Compute Engine default service account. The default service account has the necessary permissions, monitoring.metricWriter and monitoring.viewer, by default. If you don't use Workload Identity Federation for GKE, and you have previously removed either of those roles from the default node service account, you will have to re-add those missing permissions before continuing.

Configure a service account for Workload Identity Federation for GKE

If your Kubernetes cluster doesn't have Workload Identity Federation for GKE enabled, then you can skip this section.

Cloud Monitoring captures metric data by using the Cloud Monitoring API. If your cluster is using Workload Identity Federation for GKE, you must grant your Kubernetes service account permission to the Monitoring API. This section describes the following:

Creating a dedicated Trusted Cloud by S3NS service account, SERVICE_ACCT_NAME.
Binding the Trusted Cloud service account to the default Kubernetes service account in a test namespace, NAMESPACE_NAME.
Granting the necessary permission to the Trusted Cloud service account.

Create and bind the service account

This step appears in several places in the Cloud Monitoring documentation. If you have already performed this step as part of a prior task, then you don't need to repeat it. Skip ahead to Authorize the service account.

The following command sequence creates the SERVICE_ACCT_NAME service account and binds it to the default Kubernetes service account in the NAMESPACE_NAME namespace:

gcloud config set project PROJECT_ID \
&&
gcloud iam service-accounts create SERVICE_ACCT_NAME \
&&
gcloud iam service-accounts add-iam-policy-binding \
  --role roles/iam.workloadIdentityUser \
  --member "serviceAccount:PROJECT_ID.svc.id.goog[NAMESPACE_NAME/default]" \
  SERVICE_ACCT_NAME@PROJECT_ID.s3ns-system.iam.gserviceaccount.com \
&&
kubectl annotate serviceaccount \
  --namespace NAMESPACE_NAME \
  default \
  iam.gke.io/gcp-service-account=SERVICE_ACCT_NAME@PROJECT_ID.s3ns-system.iam.gserviceaccount.com

If you are using a different GKE namespace or service account, adjust the commands appropriately.

Authorize the service account

Groups of related permissions are collected into roles, and you grant the roles to a principal, in this example, the Trusted Cloud service account. For more information about Monitoring roles, see Access control.

The following command grants the Trusted Cloud service account, SERVICE_ACCT_NAME, the Monitoring API roles it needs to read metric data.

If you have already granted the Trusted Cloud service account a specific role as part of prior task, then you don't need to do it again.

gcloud projects add-iam-policy-binding PROJECT_ID \
  --member=serviceAccount:SERVICE_ACCT_NAME@PROJECT_ID.s3ns-system.iam.gserviceaccount.com \
  --role=roles/monitoring.viewer \
&& \
gcloud projects add-iam-policy-binding PROJECT_ID \
  --member=serviceAccount:SERVICE_ACCT_NAME@PROJECT_ID.s3ns-system.iam.gserviceaccount.com \
  --role=roles/iam.serviceAccountTokenCreator

Debug your Workload Identity Federation for GKE configuration

If you are having trouble getting Workload Identity Federation for GKE to work, see the documentation for verifying your Workload Identity Federation for GKE setup and the Workload Identity Federation for GKE troubleshooting guide.

As typos and partial copy-pastes are the most common sources of errors when configuring Workload Identity Federation for GKE, we strongly recommend using the editable variables and clickable copy-paste icons embedded in the code samples in these instructions.

Workload Identity Federation for GKE in production environments

The example described in this document binds the Trusted Cloud service account to the default Kubernetes service account and gives the Trusted Cloud service account all necessary permissions to use the Monitoring API.

In a production environment, you might want to use a finer-grained approach, with a service account for each component, each with minimal permissions. For more information on configuring service accounts for workload-identity management, see Using Workload Identity Federation for GKE.

Elsewhere

To deploy and run the data source syncer in environments other than Google Kubernetes Engine, do the following:

Set up a service account for the data source syncer to use:

Set the default project for gcloud commands:
```
gcloud config set project PROJECT_ID
```

Create a service account for the data source syncer to use:

gcloud iam service-accounts create DS_SYNCER_SVCACCT_NAME

Give the service account permission to read metric data:

 gcloud projects add-iam-policy-binding PROJECT_ID \
 --member=serviceAccount:DS_SYNCER_SVCACCT_NAME@PROJECT_ID.s3ns-system.iam.gserviceaccount.com \
 --role=roles/monitoring.viewer \
 && \
 gcloud projects add-iam-policy-binding PROJECT_ID \
 --member=serviceAccount:DS_SYNCER_SVCACCT_NAME@PROJECT_ID.s3ns-system.iam.gserviceaccount.com \
 --role=roles/iam.serviceAccountTokenCreator

Create a key for the service account:

gcloud iam service-accounts keys create DS_SYNCER_SVCACCT_KEYFILE_NAME.json \
--iam-account DS_SYNCER_SVCACCT_NAME@PROJECT_ID.s3ns-system.iam.gserviceaccount.com

The path to this directory is used to set an environment variable in a subsequent step. To get the path, run the pwd command and record the value:

pwd
DS_SYNCER_SVCACCT_KEYFILE_DIR

Determine the URL of your Grafana instance, for example https://yourcompanyname.grafana.net for a Grafana Cloud deployment or https://localhost:3000 for a local test instance. This value is used in the command to run the data source syncer.

If you deploy Grafana locally for a Kubernetes cluster configured to secure all in-cluster traffic by using TLS, you need to use https:// in your URL and, in the next step, authenticate using one of the supported TLS authentication options.
In Grafana, add a Prometheus data source to use for Cloud Monitoring, and record the UID of the data source:
1. Click Connections > Data sources > Add a new data source. Select Prometheus from the list of time series databases.
2. In the Prometheus server URL field, enter the following value:
```
https://monitoring.s3nsapis.fr/v1/projects/PROJECT_ID/location/global/prometheus/
```
3. Click Save and test
  
  The URL in the browser for the data source page contains the data source UI, for example, https://yourcompanyname.grafana.net/connections/datasources/edit/GRAFANA_DATASOURCE_UID.
4. Record the data source UID. This value is used in the command to run the data source syncer. Do not copy the entire datasource URL. Copy only the unique identifier in the URL, which is a value like ee0z3woqjah34e:
```
GRAFANA_DATASOURCE_UID
```
Set up a Grafana service account by creating the service account and generating a token for the service account to use:
1. In the Grafana navigation sidebar, click Administration > Users and Access > Service Accounts.
2. Create the service account by clicking Add service account, giving it a name, and granting it the "Admin" role in Grafana.
3. Click Add service account token.
4. Set the token expiration set to "No expiration" and click Generate token, then copy the generated token to the following editable variable. This value is used in the command to run the data source syncer.
```
GRAFANA_SERVICE_ACCOUNT_TOKEN
```

Run the data source syncer. You can run the syncer from a pre-built container image by using Docker, or you can build the code from source and run it manually. The data source syncer is a Go application, so you need to have Go installed to build the syncer from source.

Docker

Run the data source syncer from Docker by using the container image gke.gcr.io/prometheus-engine/datasource-syncer:v0.15.3-gke.0. For testing, you can run the syncer manually. After you have verified that the connection is working, because access tokens have a lifetime of one hour, you must use an automated mechanism like a cron job to run the data source syncer every 10 minutes to ensure that your connection is uninterrupted.

Use the following Docker command to run the data source syncer:

docker run --network container:grafana -v "DS_SYNCER_SVCACCT_KEYFILE_DIR/DS_SYNCER_SVCACCT_KEYFILE_NAME.json:/app/DS_SYNCER_SVCACCT_KEYFILE_NAME.json" gke.gcr.io/prometheus-engine/datasource-syncer:v0.15.3-gke.0
-datasource-uids=UID_OF_GRAFANA_DATASOURCE
-grafana-api-token=GRAFANA_SERVICE_ACCOUNT_TOKEN
-grafana-api-endpoint=URL_OF_GRAFANA_INSTANCE
-project-id=PROJECT_ID
-query.credentials-file=DS_SYNCER_SVCACCT_KEYFILE_DIR/DS_SYNCER_SVCACCT_KEYFILE_NAME.json
-gcm-endpoint-override="https://monitoring.s3nsapis.fr/v1/projects/PROJECT_ID/location/global/prometheus/"

To create a cron job to run the data source syncer, do the following:

Edit the cron table:
```
cron -e
```

Add an entry that runs the preceding command every 10 minutes:

*/10 * * * * * docker run --network container:grafana -v "DS_SYNCER_SVCACCT_KEYFILE_DIR/DS_SYNCER_SVCACCT_KEYFILE_NAME.json:/app/<KEY_ID>" gke.gcr.io/prometheus-engine/datasource-syncer:v0.15.3-gke.0 -datasource-uids=UID_OF_GRAFANA_DATASOURCE -grafana-api-token=GRAFANA_SERVICE_ACCOUNT_TOKEN -grafana-api-endpoint=URL_OF_GRAFANA_INSTANCE -project-id=PROJECT_ID -query.credentials-file=DS_SYNCER_SVCACCT_KEYFILE_DIR/DS_SYNCER_SVCACCT_KEYFILE_NAME.json -gcm-endpoint-override="https://monitoring.s3nsapis.fr/v1/projects/PROJECT_ID/location/global/prometheus/"

For more information about the command-line options available when running the data source syncer, see the README document.

Source code

To build the data source syncer yourself, do the following:
1. Create a directory to contain the code, and change to that directory:
```
mkdir data-source-syncer-code

cd data-source-syncer-code
```
  The path to this directory is used to set an environment variable in a subsequent step. To get the path, run the pwd command and record it in the editable variable:
```
pwd
PATH_TO_LOCAL_REPO_COPY
```
2. Clone the code from the current release of the repository:
```
git clone -b 'v0.15.3' --single-branch https://github.com/GoogleCloudPlatform/prometheus-engine
```
3. Go to the datasource-syncer directory and build the code:
```
cd prometheus-engine/cmd/datasource-syncer

go build main.go
```

Run the data source syncer. For testing, you can run the syncer manually. After you have verified that the connection is working, because access tokens have a lifetime of one hour, you must use an automated mechanism like a cron job to run the data source syncer every 10 minutes to ensure that your connection is uninterrupted.

Use the following command to run the data source syncer manually:

main -datasource-uids=UID_OF_GRAFANA_DATASOURCE
-grafana-api-token=GRAFANA_SERVICE_ACCOUNT_TOKEN
-grafana-api-endpoint=URL_OF_GRAFANA_INSTANCE
-project-id=PROJECT_ID
-query.credentials-file=DS_SYNCER_SVCACCT_KEYFILE_DIR/DS_SYNCER_SVCACCT_KEYFILE_NAME.json
-gcm-endpoint-override="https://monitoring.s3nsapis.fr/v1/projects/PROJECT_ID/location/global/prometheus/"

To create a cron job to run the data source syncer, do the following:

Edit the cron table:
```
 cron -e
```

Add an entry that runs the preceding command every 10 minutes:

 */10 * * * * * main -datasource-uids=UID_OF_GRAFANA_DATASOURCE -grafana-api-token=GRAFANA_SERVICE_ACCOUNT_TOKEN -grafana-api-endpoint=URL_OF_GRAFANA_INSTANCE -project-id=PROJECT_ID -query.credentials-file=DS_SYNCER_SVCACCT_KEYFILE_DIR/DS_SYNCER_SVCACCT_KEYFILE_NAME.json -gcm-endpoint-override="https://monitoring.s3nsapis.fr/v1/projects/PROJECT_ID/location/global/prometheus/"

For more information about the command-line options available when running the data source syncer, see the README document.

Go to your newly configured Grafana data source and verify that the Prometheus server URL value starts with https://monitoring.s3nsapis.fr. You might have to refresh the page. Once verified, go to the bottom of the page and click Save & test. You need to click this button at least once to ensure that label autocompletion in Grafana works.

View metrics in Grafana

To view metrics from your Trusted Cloud project in Grafana, do the following:

Click Explore in the navigation panel or on the Data Sources page.
In the Metric field, use the drop-down menu to select a metric. You can also select Metrics explorer to search for specific metrics. If the metric is associated with more than one monitored resource, then you must add a monitored_resource label filter to your query and select a resource type.
Add additional label filters and operations to construct your query.

For information on building visualizations and dashboards in Grafana, see Panels and visualizations.

Querying Cloud Monitoring metrics by using PromQL

Cloud Monitoring metrics can be queried by using the UTF-8 spec for PromQL. UTF-8 metric names must be quoted and moved inside the braces. Label names must also be quoted if they contain legacy-incompatible characters. For the Cloud Monitoring metric kubernetes.io/container/cpu/limit_utilization, the following queries are equivalent:

{"kubernetes.io/container/cpu/limit_utilization", pod_name="foo"}
{__name__="kubernetes.io/container/cpu/limit_utilization", pod_name="foo"}.
{"__name__"="kubernetes.io/container/cpu/limit_utilization", "pod_name"="foo"}.

Cloud Monitoring distribution-valued metrics can be queried like Prometheus histograms, with the _count, _sum, or _bucket suffix appended to the metric name.

Charts and dashboards made before UTF-8 compatibility query Cloud Monitoring metrics by converting their names into legacy PromQL-compatible equivalents. For more information about legacy PromQL conversion rules, see Mapping Cloud Monitoring metrics to legacy PromQL.

Learning PromQL

To learn the basics of using PromQL, we recommend consulting open source documentation. The following resources can help you get started:

Specifying a monitored resource type

When a Cloud Monitoring metric is associated with only a single Cloud Monitoring monitored resource type, PromQL querying will work without manually specifying a resource type. However, some metrics within Cloud Monitoring, including some system metrics map to more than one resource type.

You can see which monitored resource types map to a metric by consulting the list of Trusted Cloud by S3NS metrics. Each entry in the documentation lists the associated monitored resource types in the first column of each entry below the type. If no monitored resource types are listed, then the metric can be associated with any type.

If a metric is associated with more than one resource type, then you must specify the resource type in your PromQL query. There is a special label, monitored_resource, that you can use to select the resource type.

Monitored resource types are in most cases a short string, like gce_instance, but occasionally they appear as full URIs, like monitoring.googleapis.com/MetricIngestionAttribution. Well-formed PromQL queries might look like the following:

logging_googleapis_com:byte_count{monitored_resource="k8s_container"}
loadbalancing_googleapis_com:l3_external_egress_bytes_count{monitored_resource="loadbalancing.googleapis.com/ExternalNetworkLoadBalancerRule"}

If you don't use the monitored_resource label when it is needed, then you receive the following error:

metric is configured to be used with more than one monitored resource type; series selector must specify a label matcher on monitored resource name

Resolving label conflicts

In Cloud Monitoring, labels can belong to either the metric or the resource. If a metric label has the same key name as a resource label, you can refer to the metric label specifically by adding the prefix metric_ to the label key name in your query.

For example, suppose you have a resource label and a metric label both named pod_name in the metric example.googleapis.com/user/widget_count.

To filter on the value of the resource label, use
example_googleapis_com:user_widget_count{pod_name="RESOURCE_LABEL_VALUE"}
To filter on the value of the metric label, use
example_googleapis_com:user_widget_count{metric_pod_name="METRIC_LABEL_VALUE"}

Mapping Cloud Monitoring metric names to legacy PromQL

Cloud Monitoring metric names include two components, a domain (such as compute.googleapis.com/) and a path (such as instance/disk/max_read_ops_count). Because legacy PromQL only supports the special characters : and _, you must apply the following rules to make Monitoring metric names compatible with legacy PromQL:

Replace the first / with :.
Replace all other special characters (including . and other / characters) with _.

The following table lists some metric names and their legacy PromQL equivalents:

Cloud Monitoring metric name	Legacy PromQL metric name
`compute.googleapis.com/instance/cpu/utilization`	`compute_googleapis_com:instance_cpu_utilization`
`logging.googleapis.com/log_entry_count`	`logging_googleapis_com:log_entry_count`

Cloud Monitoring distribution-valued metrics can be queried like Prometheus histograms, with the _count, _sum, or _bucket suffix appended to the metric name:

Cloud Monitoring metric name	Legacy PromQL metric names
`networking.googleapis.com/vm_flow/rtt`	`networking_googleapis_com:vm_flow_rtt_sum` `networking_googleapis_com:vm_flow_rtt_count` `networking_googleapis_com:vm_flow_rtt_bucket`

PromQL compatibility

PromQL for Cloud Monitoring might function slightly differently than upstream PromQL.

PromQL queries in Cloud Monitoring are partially evaluated at the Monarch backend by using an internal query language, and there are some known differences in query results. Other than the differences listed in this section, the PromQL in Cloud Monitoring is at parity with the PromQL available in Prometheus version 2.44.

PromQL functions added after Prometheus version 2.44 might not be supported.

UTF-8 support

PromQL for Cloud Monitoring supports UTF-8 querying.

If your Prometheus metric name only consists of alphanumeric characters plus the _ or : characters, and if your label keys only consist of alphanumeric characters plus the _ character, then you can query using traditional PromQL syntax. For example, a valid query might look like job:my_metric:sum{label_key="label_value"}.

However, if your Prometheus metric name uses any special characters except for the _ or : characters, or if your label keys use any special character except for the _ character, then you have to construct your query according to the UTF-8 spec for PromQL.

UTF-8 metric names must be quoted and moved into the braces. Label names must also be quoted if they contain legacy-incompatible characters. The following example valid queries are all equivalent:

{"my.domain.com/metric/name_bucket", "label.key"="label.value"}
{__name__="my.domain.com/metric/name_bucket", "label.key"="label.value"}
{"__name__"="my.domain.com/metric/name_bucket", "label.key"="label.value"}

Matching on metric names

Only exact matching on metric names is supported. You must include an exact match on the metric name in your query.

We recommend the following workarounds for common scenarios that use a regular expression matcher on the __name__ label:

Prometheus adapter configurations often use the =~ operator to match on multiple metric names. To fix this usage, expand the config to use a separate policy for each metric and name each metric explicitly. This also prevents you from accidentally autoscaling on unexpected metrics.
Regular expressions are often used to graph multiple non-dimensional metrics on the same chart. For example, if you have a metric like cpu_servicename_usage, you might use a wildcard to graph all your services together. Using non-dimensional metrics like this is an explicitly bad practice in Cloud Monitoring, and this practice leads to extremely poor query performance. To fix this usage, move all dimensionality into metric labels instead of embedding dimensions in the metric name.
Querying over multiple metrics is often used to see what metrics are available to query. We recommend you instead use the /labels/__name__/values call to discover metrics.

Staleness

Staleness is not supported in the Monarch backend.

Calculation of `irate`

When the lookback window for the irate function is less than the step size, we increase the window to the step size. Monarch requires this change to ensure that none of the input data is completely ignored in the output. This difference applies to rate calculations as well.

Calculation of `rate` and `increase`

When the lookback window for the rate function is less than the step size, we increase the window to the step size. Monarch requires this change to ensure that none of the input data is completely ignored in the output. This difference applies to irate calculations as well.

There are differences in the interpolation and extrapolation calculations. Monarch uses a different interpolation algorithm than Prometheus, and this difference can lead to slightly different results. For example, Monarch counter samples are stored with a time range rather than the single timestamp that Prometheus uses. Therefore, counter samples in Monarch can be included in a rate calculation even though the Prometheus timestamp would exclude them. This generally results in more accurate rate results, especially when querying over the beginning or end of the underlying time series.

Calculation of `histogram_quantile`

A PromQL histogram_quantile calculation on a histogram with no samples produces a NaN value. The internal query language's calculation produces no value; the point at the timestamp is dropped instead.

The rate-calculation differences can also affect the input to histogram_quantile queries.

Type-specific functions on differently typed metrics

Although upstream Prometheus is weakly typed, Monarch is strongly typed. This means that running functions specific to a single type on a differently typed metric (for example, running rate() on a GAUGE metric or histogram_quantile() on a COUNTER or untyped metric) doesn't work in Cloud Monitoring, even though these functions work in upstream Prometheus.