Use the built-in Prometheus server

When you install Gloo, a Prometheus server is automatically set up and configured to scrape metrics from the Gloo management server.

Prometheus records and collects multi-dimensional data over time. With this data, you can easily see important health-related information for your service mesh, such as which routes perform well in your service mesh, where you might have a bottleneck, how fast your services respond to requests, or how many requests per second the services process. All data is stored in the Prometheus database and you can use the Prometheus Querying Language (PromQL) to perform complex queries and monitor how metrics change over time. In addition, you can easily set up alerts for when metrics reach a certain threshold.

For more information about the built-in Prometheus server, the default setup and an overview of metrics that are available to you, see Gloo metrics architecture.

On this page:

Prometheus annotations are automatically added during the Istio installation to enable scraping of metrics for the Istio control plane (istiod), ingress, and proxy pods. These metrics are automatically merged with app metrics and made available to the Gloo management server. For more information about these annotations and how you can disable them, see the Istio documentation.

Open the built-in Prometheus dashboard

If you used the the default Helm chart values when you installed Gloo Enterprise, the Prometheus server is automatically set up for you in the Gloo management cluster.

The information in this guide assumes that you followed the getting started tutorials to install Gloo, install Istio with the IstioOperator manifest in the workload clusters, and deploy the Bookinfo app.

  1. Check if the Prometheus server is running in your Gloo management cluster.

    kubectl get pods -n gloo-mesh --context $MGMT_CONTEXT | grep prometheus 

    Example output:

    prometheus-server-647b488bb-wxlzh        2/2     Running   0          66m

    If no Prometheus server is set up in your management cluster, you can enable the Prometheus server by upgrading your Helm chart with the following command.

    helm upgrade gloo-mgmt gloo-mesh-enterprise/gloo-mesh-enterprise \
    --namespace gloo-mesh \
    --set prometheus.enabled=true \
    --kube-context $MGMT_CONTEXT \
    --version $GLOO_VERSION \
    --values values-mgmt-plane-env.yaml

    Make sure to include your Helm values when you upgrade either as a configuration file in the –values flag or with –set flags. Otherwise, any previous custom values that you set might be overwritten. In single cluster setups, this might mean that your Gloo agent and ingress gateways are removed. For more information, see Get your Helm chart values in the upgrade guide.

  2. Set up port forwarding on your local machine to access the Prometheus dashboard.

    kubectl -n gloo-mesh port-forward deploy/prometheus-server 9090 --context $MGMT_CONTEXT
  3. In your web browser, open the Prometheus dashboard in your web browser: localhost:9090/

View service mesh metrics in Prometheus

Use the product page microservice from the Bookinfo app to test how metrics are sent from the Envoy proxy to see metrics that the Gloo agent collects for all the Envoy proxies in your service mesh.

The information in this guide assumes that you followed the getting started tutorials to install Gloo, install Istio with the IstioOperator manifest in the workload clusters, and deploy the Bookinfo app.

  1. Set up port forwarding for the product page of the Istio Bookinfo app.

    kubectl -n bookinfo port-forward deploy/productpage-v1 9080 --context $REMOTE_CONTEXT1
  2. Send multiple requests to the product page. Each request to the product page requires the product page to collect data from other Bookinfo microservices. For each request, metrics are automatically sent to the Prometheus server.

    for ((i=1;i<=10;i++)); do curl -I -k "http://localhost:9080/productpage"; done
  3. From the Prometheus dashboard, enter the following PromQL query to see how many requests the product page's Envoy proxy in cluster1 sent to other Bookinfo microservices.

    sum(istio_requests_total{source_workload="productpage-v1"}) by (workload_id,destination_workload_id, response_code)
  4. Explore other queries that you run in the Prometheus dashboard to gain insight into your service mesh.

Sample queries to monitor apps in your service mesh

You can use the following metrics in your Prometheus UI to gain insight into the performance and health of the apps that run in your service mesh.

Metric PromQL query
Request rate for a given service rate(istio_requests_total{destination_app="<service_ID>"}[5m])
Request rate between source and destination workload rate(istio_requests_total{source_workload="<source_workload>", destination_workload="<workload_ID>"}[5m])
Successful request rate to a destination workload rate(istio_requests_total{response_code=~"[2-3].*", destination_workload="<workload_ID>"}[5m])
Rate of failing requests to a destination workload rate(istio_requests_total{response_code=~"[4-5].*", destination_workload="<workload_ID>"}[5m])
Number of new requests within a certain timeframe sum(increase(istio_requests_total{}[5m])) by (workload_id, destination_workload_id)