Gloo Metrics

All Gloo pods ship with optional Prometheus monitoring capabilities.

This functionality is turned off by default, and can be turned on a couple of different ways: through Helm chart install options; and through environment variables.

Once you have enabled your metrics, you can take a look at the Help strings we publish to see what kind of metrics are available.

Enabling Pod Metrics

Helm Chart Options

The first way is via the helm chart. All deployment objects in the helm templates accept an argument stats which when set to true, start a stats server on the given pod.

For example, to add stats to the Gloo gateway, when installing with Helm add --set discovery.deployment.stats=true.

helm install gloo/gloo \
  --name gloo \
  --namespace gloo-system \
  --set discovery.deployment.stats=true

Here’s what the resulting discovery manifest would look like. Note the additions of the prometheus.io annotations, and the START_STATS_SERVER environment variable.

apiVersion: extensions/v1beta1
kind: Deployment
metadata:
  labels:
    app: gloo
    gloo: discovery
  name: discovery
  namespace: gloo-system
spec:
  replicas: 1
  selector:
    matchLabels:
      gloo: discovery
  template:
    metadata:
      labels:
        gloo: discovery
      annotations:
        prometheus.io/path: /metrics
        prometheus.io/port: "9091"
        prometheus.io/scrape: "true"
    spec:
      containers:
      - image: "quay.io/solo-io/discovery:0.11.1"
        imagePullPolicy: Always
        name: discovery
        env:
          - name: POD_NAMESPACE
            valueFrom:
              fieldRef:
                fieldPath: metadata.namespace
          - name: START_STATS_SERVER
            value: "true"

This flag will set the START_STATS_SERVER environment variable to true in the container which will start the stats server on port 9091.

Environment Variables

The other method is to manually set the START_STATS_SERVER=1 in the pod.

Monitoring Gloo with Prometheus

Prometheus has great support for monitoring kubernetes pods. Docs for that can be found here. If the stats are enabled through the Helm chart than the Prometheus annotations are automatically added to the pod spec. And those Prometheus stats are available from the admin page in our pods.

For example, assuming you installed Gloo as previously using Helm, and enabled stats for discovery, you could then kubectl port-forward <pod> 9091:9091 those pods (or deployments/services selecting those pods) to access their admin page as follows.

kubectl --namespace gloo-system port-forward deployment/discovery 9091:9091

And then open http://localhost:9091 for the admin page, including the Prometheus metrics at http://localhost:9091/metrics.

More information on Gloo’s admin ports can be found here.

Metrics Context

You can see exactly what metrics are published from a particular pod by taking a look at our Prometheus Help strings. For a given pod you’re interested in, you can curl /metrics on its stats port (usually 9091) to see this content.

For example, here’s a look at the Help strings published by our gloo pod as of 0.20.13. You can do the same thing for any of our pods, including the closed-source ones in the case of Gloo Enterprise.

$ kubectl port-forward deployment/gloo 9091 &
$ portForwardPid=$!
$ curl -s localhost:9091/metrics | grep HELP
# HELP api_gloo_solo_io_emitter_resources_in The number of resource lists received on open watch channels
# HELP api_gloo_solo_io_emitter_snap_in Deprecated. Use api.gloo.solo.io/emitter/resources_in. The number of snapshots updates coming in.
# HELP api_gloo_solo_io_emitter_snap_missed The number of snapshots updates going missed. this can happen in heavy load. missed snapshot will be re-tried after a second.
# HELP api_gloo_solo_io_emitter_snap_out The number of snapshots updates going out
# HELP eds_gloo_solo_io_emitter_resources_in The number of resource lists received on open watch channels
# HELP eds_gloo_solo_io_emitter_snap_in Deprecated. Use eds.gloo.solo.io/emitter/resources_in. The number of snapshots updates coming in.
# HELP eds_gloo_solo_io_emitter_snap_out The number of snapshots updates going out
# HELP gloo_solo_io_setups_run The number of times the main setup loop has run
# HELP grpc_io_server_completed_rpcs Count of RPCs by method and status.
# HELP grpc_io_server_received_bytes_per_rpc Distribution of received bytes per RPC, by method.
# HELP grpc_io_server_received_messages_per_rpc Distribution of messages received count per RPC, by method.
# HELP grpc_io_server_sent_bytes_per_rpc Distribution of total sent bytes per RPC, by method.
# HELP grpc_io_server_sent_messages_per_rpc Distribution of messages sent count per RPC, by method.
# HELP grpc_io_server_server_latency Distribution of server latency in milliseconds, by method.
# HELP kube_events_count The number of events sent from kuberenets to us
# HELP kube_lists_count The number of list calls
# HELP kube_req_in_flight The number of requests in flight
# HELP kube_updates_count The number of update calls
# HELP kube_watches_count The number of watch calls
# HELP runtime_goroutines The number of goroutines
# HELP setup_gloo_solo_io_emitter_resources_in The number of resource lists received on open watch channels
# HELP setup_gloo_solo_io_emitter_snap_in Deprecated. Use setup.gloo.solo.io/emitter/resources_in. The number of snapshots updates coming in.

$ kill $portForwardPid