Default Envoy tracing

With Gloo Gateway, you can use Envoy’s end-to-end tracing capabilities to obtain visibility and track requests as they pass through your API gateway to distributed backends, such as services, databases, or other endpoints in your ecosystem.

To get started, enable the default Envoy distributed tracing in your Gloo Gateway installation to trace requests, analyze service dependencies, and find bottlenecks or high latency services. Then, use a tracing provider to collect and visualize the sampled spans. The following distributed tracing platforms are supported for the default Envoy metrics collection in Gloo Gateway:

This guide uses the Zipkin tracing platform as an example to show how to set up tracing in Gloo Gateway. To set up other tracing platforms, refer to the platform-specific documentation or the Envoy tracing provider documentation.

How does it work?

To trace a request, data must be captured from the moment the request is initiated and every time the request is forwarded to another endpoint, or when other microservices are called along the way. When a request is initiated, a trace ID and an initial span (parent span) is created. A span represents an operation that is performed on your request, such as an API call, a database lookup, or a call to an external service. If a request is sent to a service, a child span is created in the trace capturing all the operations that are performed within the service.

Each operation and span is documented with a timestamp so that you can easily see how long a request was processed by a specific endpoint in your trace. Most tracing platforms have support to visualize the tracing information in a graph so that you can easily see bottlenecks in your microservices stack.

To configure a tracing platform, you must update the Envoy bootstrap configuration. The bootstrap configuration is automatically applied when an Envoy process is initialized. To update the bootstrap configuration, you can use one of the following ways:

Set up Zipkin tracing

To set up Zipkin tracing, you follow these general steps:

  1. Set up Zipkin locally.
  2. Configure the Zipkin tracing cluster in Gloo Gateway.
  3. Configure Zipkin as the tracing provider for a listener.
  4. Optional: Annotate routes with descriptors.
  5. Initiate a request and view traces.

Step 1. Set up Zipkin locally

Set up Zipkin tracing in a local Kind cluster for local troubleshooting and experimentation.

  1. Run Zipkin.

    docker run --network=kind -itd --name zipkin -p 9411:9411 openzipkin/zipkin
    
  2. Verify that both zipkin and zipkin-tracing-control-plane are in your local Kind cluster network, and note the zipkin container IP address without the CIDR. In the following example, the zipkin container IP address is 172.xx.x.2.

    docker network inspect kind
    
       [
           {
               "Name": "kind",
               "Id": "6a37a4ebb2d0e7dcbabe50dc8b1a519b431f054aebb822ed85e00abde99fd4d3",
               "Created": "2021-09-16T09:28:49.88165506-04:00",
               "Scope": "local",
               "Driver": "bridge",
               "EnableIPv6": true,
               "IPAM": {
                   "Driver": "default",
                   "Options": {},
                   "Config": [
                       {
                           "Subnet": "172.xx.x.0/16",
                           "Gateway": "172.xx.x.1"
                       },
                       {
                           "Subnet": "fc00:f853:ccd:e793::/64",
                           "Gateway": "fc00:f853:ccd:e793::1"
                       }
                   ]
               },
               "Internal": false,
               "Attachable": false,
               "Ingress": false,
               "ConfigFrom": {
                  "Network": ""
               },
               "ConfigOnly": false,
               "Containers": {
                   "3431770d0c41bfbc8eceac4c806605286f5dac81820599f828dcb250037a2f48": {
                       "Name": "zipkin-tracing-control-plane",
                       "EndpointID": "3e48e18bc7b259ca9d597a594ee3d5205c8339e8ecd9f8f274a178d07f395b78",
                       "MacAddress": "02:42:ac:12:00:03",
                       "IPv4Address": "172.xx.x.3/16",
                       "IPv6Address": "fc00:f853:ccd:e793::3/64"
                   },
                   "84dadbd86f113c7104eca23d3d78e9dec997a47666c1ba4eed2ae7a5ad8eb20d": {
                       "Name": "zipkin",
                       "EndpointID": "09e07c8ac6b1cd912c325962586d9497520e216a4ec357384c663594248fc104",
                       "MacAddress": "02:42:ac:12:00:02",
                       "IPv4Address": "172.xx.x.2/16",
                       "IPv6Address": "fc00:f853:ccd:e793::2/64"
                   }
               },
               "Options": {
                   "com.docker.network.bridge.enable_ip_masquerade": "true",
                   "com.docker.network.driver.mtu": "1500"
               },
               "Labels": {}
           }
       ]
       
  3. Configure the Zipkin tracing cluster with the IP address that was assigned in the previous step. In this example, the Zipkin cluster is assigned the 172.18.0.2 IP address.

  4. Open Zipkin on your local machine.

    Zipkin UI

Step 2. Configure the Zipkin tracing cluster in Gloo Gateway

Zipkin uses a dedicated tracing cluster where tracing information is sent to. The name of the tracing cluster must be set in the Envoy bootstrap configuration for Envoy to know where to send the information to. The following example shows how you can configure the Zipkin tracing cluster by using Gloo Gateway or updating the Envoy bootstrap configuration directly.

Option 1: Install Gloo Gateway with Zipkin tracing

Use the Gloo Gateway installation Helm chart template to configure the Zipkin tracing platform. Gloo Gateway automatically determines the updates that must be made to apply the Zipkin configuration in your Envoy proxies.

  1. Create a values.yaml file and add your Zipkin configuration. In the following example, the Zipkin cluster is called zipkin. Replace <zipkin_container_IP_address> with the value that you retrieved in Step 1: Set up Zipkin locally, such as 172.xx.x.2.

       gatewayProxies: 
         gatewayProxy:
           tracing:
             cluster:
               - name: zipkin
                 connect_timeout: 1s
                 type: STRICT_DNS
                 load_assignment:
                   cluster_name: zipkin
                   endpoints:
                   - lb_endpoints:
                     - endpoint:
                         address:
                           socket_address:
                             address: <zipkin_container_IP_address>
                             port_value: 9411
       
  2. Install Gloo Gateway with your Zipkin configuration.

    helm install gloo gloo/gloo --namespace gloo-system --create-namespace -f values.yaml
    

Option 2: Update the Envoy configmap directly

Add the Envoy code that you want to apply to a Kubernetes configmap and restart the proxy deployments.

  1. Edit the Envoy proxy configuration.

    kubectl edit configmap -n gloo-system gateway-proxy-envoy-config
    
  2. Enter the Zipkin tracing changes. Replace <zipkin_container_IP_address> with the value that you retrieved in Step 1: Set up Zipkin locally, such as 172.xx.x.2.

       apiVersion: v1
       kind: ConfigMap
       data:
         envoy.yaml:
           node:
             cluster: gateway
             id: "{{.PodName}}{{.PodNamespace}}"
             metadata:
               role: "{{.PodNamespace}}~gateway-proxy"
           static_resources:
             clusters:
               - name: xds_cluster
                 connect_timeout: 5.000s
                 load_assignment:
                   cluster_name: xds_cluster
                   endpoints:
                   - lb_endpoints:
                     - endpoint:
                         address:
                           socket_address:
                             address: gloo
                             port_value: 9977
                 http2_protocol_options: {}
                 type: STRICT_DNS
               - name: zipkin
                 connect_timeout: 1s
                 type: STRICT_DNS
                 load_assignment:
                   cluster_name: zipkin
                   endpoints:
                   - lb_endpoints:
                     - endpoint:
                         address:
                           socket_address:
                             address: <zipkin_container_IP_address>
                             port_value: 9411
       
  3. Apply the updated Envoy config. For Envoy to pick up the new config, you need to restart the Envoy proxy deployment.

    kubectl rollout restart deployment gateway-proxy
    

Step 3. Configure Zipkin as the tracing provider for a listener

After you configure the tracing cluster, you can now set Zipkin as the tracing platform for a listener in your gateway. To do that, you can either update the gateway or provide the Envoy code in a Kubernetes configmap. Then, apply this configmap by manually restarting the Envoy proxies.

When you choose to manually update the Envoy proxies with a configmap, you can apply the updated configuration to a static listener that is defined in the Envoy bootstrap config only. If you want to configure a tracing provider for dynamically created listeners, you must update the gateway.

Option 1: Dynamic listeners with Gloo Gateway

You can enable tracing on a listener-by-listener basis. To find an example tracing listener configuration for your gateway, see the tracing listener docs. In this example, the Zipkin cluster that you created in step 1 is referenced in the clusterName field.

Option 2: Static listeners with configmaps

  1. Edit the Envoy proxy configuration.

    kubectl edit configmap -n gloo-system gateway-proxy-envoy-config
    
  2. Enter the tracing provider changes.

       apiVersion: v1
       kind: ConfigMap
       data:
         envoy.yaml:
           node:
             cluster: gateway
             id: "{{.PodName}}{{.PodNamespace}}"
             metadata:
               role: "{{.PodNamespace}}~gateway-proxy"
           static_resources:
             listeners:
               - name: prometheus_listener
                 address:
                   socket_address:
                     address: 0.0.0.0
                     port_value: 8081
                 filter_chains:
                   - filters:
                       - name: envoy.filters.network.http_connection_manager
                         typed_config:
                           "@type": type.googleapis.com/envoy.extensions.filters.network.http_connection_manager.v3.HttpConnectionManager
                           codec_type: AUTO
                           stat_prefix: prometheus
                           route_config: # collapsed for brevity
                           http_filters:
                             - name: envoy.filters.http.router
                           tracing:
                             provider:
                               name: envoy.tracers.zipkin
                               typed_config:
                                 "@type": "type.googleapis.com/envoy.config.trace.v3.ZipkinConfig"
                                 collector_cluster: zipkin
                                 collector_endpoint: "/api/v2/spans"
                                 collector_endpoint_version: HTTP_JSON
       
  3. Apply the updated Envoy config. For Envoy to pick up the new config, you need to restart the Envoy proxy deployment.

    kubectl rollout restart deployment [deployment_name]
    

This provider configuration will only be applied to the static listeners that are defined in the bootstrap config. If you need to support tracing on dynamically created listeners, see Option 1: Dynamic listeners with Gloo Gateway.

Step 4. Optional: Annotate routes with tracing descriptors

In order to associate a trace with a route, it can be helpful to annotate your routes with a descriptive name. You can add the name to the virtual service directly, or use the x-envoy-decorator-operation Envoy header in your request. If a name is set in both, the name in the header takes precedence.

The following steps show how to add the name to the virtual service in Gloo Gateway.

  1. List the virtual services in the gloo-system namespace and select the one that you want to edit.

    kubectl get virtualservice -n gloo-system
    
  2. Edit the virtual service.

    kubectl edit virtualservice -n gloo-system <virtual-service-name>
    
  3. Enter the name for the route that you want to associate your trace with in the routeDescriptor field.

       apiVersion: gateway.solo.io/v1
       kind: VirtualService
       metadata: # omitted for brevity
       spec:
         virtualHost:
           domains:
           - '*'
           routes:
           - matchers:
             - exact: /abc
             routeAction:
               single:
                 upstream:
                   name: my-upstream
                   namespace: gloo-system
             options:
               tracing:
                 routeDescriptor: <route-descriptor-name>
               prefixRewrite: /
       status: # omitted for brevity
       

Step 5. Initiate a request and view traces

  1. Send a request to your app.

    curl localhost:31500/abc
    
  2. Open Zipkin on your local machine and review the traces.