Debug logging for transformations

You can use Gloo Gateway’s debug logging feature to debug complex sequences of transformations.

This feature has the potential to log sensitive information. Do not enable this feature in production environments.

Before you begin

This guide assumes that you installed the following components:

You also need an upstream service to serve as the target for the requests that you send to test the Gloo Gateway configurations in this tutorial. You can use the publicly available Postman Echo service. Postman Echo exposes a set of endpoints that are very useful for inspecting both the requests sent upstream and the resulting responses. For more information about this service, see the Postman Echo documentation.

Create a static upstream to represent the postman-echo.com remote service.

apiVersion: gloo.solo.io/v1
kind: Upstream
metadata:
  name: postman-echo
  namespace: gloo-system
spec:
  static:
    hosts:
    - addr: postman-echo.com
      port: 80

Next, create a simple Virtual Service that matches any path and routes all traffic to the Upstream:


apiVersion: gateway.solo.io/v1
kind: VirtualService
metadata:
  name: test-debug-logs
  namespace: gloo-system
spec:
  virtualHost:
    domains:
    - '*'
    routes:
    - matchers:
       - prefix: /
      routeAction:
        single:
          upstream:
            name: postman-echo
            namespace: gloo-system

Test that the configuration was correctly picked up by Gloo Gateway by executing the following command:

curl $(glooctl proxy url)/get | jq

Example output:

{
  "args": {},
  "headers": {
    "x-forwarded-proto": "https",
    "host": "postman-echo.com",
    "accept": "*/*",
    "user-agent": "curl/7.54.0",
    "x-envoy-expected-rq-timeout-ms": "15000",
    "x-request-id": "bd20e4be-3c8a-405f-80a4-027204f732cb",
    "x-forwarded-port": "80"
  },
  "url": "https://postman-echo.com/get"
}

Add debug logging to a transformation

You can add debug logging to individual transformations in your Virtual Service.

  1. Update the virtual service to include transformations that add headers to the response.

    apiVersion: gateway.solo.io/v1
    kind: VirtualService
    metadata:
      name: test-debug-logs
      namespace: gloo-system
    spec:
      virtualHost:
        domains:
        - '*'
        routes:
        - matchers:
           - prefix: /
          routeAction:
            single:
              upstream:
                name: postman-echo
                namespace: gloo-system
        options:
          stagedTransformations:
            early:
              requestTransforms:
              - requestTransformation:
                  transformationTemplate:
                    headers:
                      x-early-request-header: 
                        text: "early"
            regular:
              requestTransforms:
              - requestTransformation:
                  transformationTemplate:
                    headers:
                      x-regular-request-header: 
                        text: "regular"
       
  2. Test that Gloo Gateway picked up the configuration update.

    curl $(glooctl proxy url)/get | jq
    

    Example output: Notice that the response now includes the early and regular headers that you added in the transformation.

    {
      "args": {},
      "headers": {
        "x-forwarded-proto": "http",
        "x-forwarded-port": "80",
        "host": "postman-echo.com",
        "x-amzn-trace-id": "Root=1-64886c58-5abcc9740f12068f5d1fabc8",
        "user-agent": "curl/7.87.0",
        "accept": "*/*",
        "x-request-id": "7342aa2e-77ed-4092-a646-1793ece6aab0",
        "x-early-request-header": "early",
        "x-regular-request-header": "regular",
        "x-envoy-expected-rq-timeout-ms": "15000"
      },
      "url": "http://postman-echo.com/get"
    }
       
  3. Add debug logging by using the logRequestResponseInfo setting. Note that logging is added only to the regular stage of the transformation, not the early stage.

    apiVersion: gateway.solo.io/v1
    kind: VirtualService
    metadata:
      name: test-debug-logs
      namespace: gloo-system
    spec:
      virtualHost:
        domains:
        - '*'
        routes:
        - matchers:
           - prefix: /
          routeAction:
            single:
              upstream:
                name: postman-echo
                namespace: gloo-system
        options:
          stagedTransformations:
            early:
              requestTransforms:
              - requestTransformation:
                  transformationTemplate:
                    headers:
                      x-early-request-header: 
                        text: "early"
            regular:
              requestTransforms:
              - requestTransformation:
                  logRequestResponseInfo: true
                  transformationTemplate:
                    headers:
                      x-regular-request-header: 
                        text: "regular"
       
  4. Set the log level of the gateway-proxy pod to debug so that you can view debug logs.

    kubectl port-forward -n gloo-system deployment/gateway-proxy 19000:19000
    curl "localhost:19000/logging?level=debug" -X POST
    
  5. Repeat the request to generate fresh logs. The response matches your previous output.

    curl $(glooctl proxy url)/get | jq
    
  6. Check the debug logs in the gateway-proxy.

    kubectl logs -n gloo-system deployment/gateway-proxy
    

    Example output: The body and request headers are logged before and after the transformation. Note that only the x-regular-request-header is present in the request headers after the transformation is processed, because the regular transformation is the one that you enabled debug logging for.

    [2023-06-13 13:28:27.724][38][debug][filter] [source/extensions/filters/http/transformation/transformation_filter.cc:257] [C4][S8942229055955075319] headers before transformation: ':authority', 'localhost:8080'
    ':path', '/get'
    ':method', 'GET'
    ':scheme', 'http'
    'user-agent', 'curl/7.87.0'
    'accept', '*/*'
    'x-forwarded-proto', 'http'
    'x-request-id', '18a9cdc4-c2e4-4255-84a6-5e69f4ef0ca2'
    'x-early-request-header', 'early'
    
    [2023-06-13 13:28:27.724][38][debug][filter] [source/extensions/filters/http/transformation/transformation_filter.cc:259] [C4][S8942229055955075319] body before transformation: 
    [2023-06-13 13:28:27.724][38][debug][filter] [source/extensions/filters/http/transformation/transformation_filter.cc:263] [C4][S8942229055955075319] headers after transformation: ':authority', 'localhost:8080'
    ':path', '/get'
    ':method', 'GET'
    ':scheme', 'http'
    'user-agent', 'curl/7.87.0'
    'accept', '*/*'
    'x-forwarded-proto', 'http'
    'x-request-id', '18a9cdc4-c2e4-4255-84a6-5e69f4ef0ca2'
    'x-early-request-header', 'early'
    'x-regular-request-header', 'regular'
    

Notes

You can log request and response information at the following levels:

Cleanup

Clean up the resources created in this tutorial:

kubectl delete virtualservice -n gloo-system test-debug-logs
kubectl delete upstream -n gloo-system postman-echo