After applying a policy, you might notice unexpected behavior or conversely no effect on your traffic.

Before you begin, you can review the Policy enforcement concepts to understand more about how policies are applied, import and export workspace considerations, and supported policy for type of gateway.

  1. Check your Gloo policy resources in the Gloo UI or your terminal.

  2. In your policy details, check for common issues, such as the following:

    • Any configuration error messages
    • Underlying servers that must exist, such as the rate limit or external auth server configurations for those policies
    • Correctly selected routes, destinations, or workloads
    • Correctly imported or exported resources
    • Typos in resource labels, names, namespaces
  3. Confirm that the policy selects the correct, healthy upstream service.

  4. Check the underlying Istio and Envoy resources that set up routing behavior in your Gloo environment. For Gloo Mesh Gateway, these resources are typically in the same namespace as your ingress gateway (such as gloo-mesh-gateways). For Gloo Mesh, these resources can be in all of the namespaces in your service mesh.

    1. Check the Istio virtual service for the host and route that you previously identified.
        kubectl get virtualservice -A
      kubectl get virtualservice <virtual-service-name> -n gloo-mesh-gateways -o yaml
        
      In the output, verify that the host, route, service, and gateway all match what you expect.
        
      gateways:
      - <virtual-gateway-name>
      hosts:
      - api.example.com
      http:
      - match:
        - sourceLabels:
            app: istio-ingressgateway
            istio: ingressgateway
            revision: 1-18
          uri:
            prefix: /trackapi/
        name: tracks-api-tracks-rt.gloo-mesh-gateways.cluster1-portal--api-example-com-rt.gloo-mesh-gateways.cluster1-portal
        rewrite:
          uri: /
        route:
        - destination:
            host: tracks-rest-api.tracks.svc.cluster.local
            port:
              number: 5000
        
    2. Check the matching Istio gateway.
        kubectl get gateways -A
      kubectl get gateway <virtual-gateway-name> -n gloo-mesh-gateways -o yaml
        
      In the output, confirm that the gateway, host, and protocol all match what you expect.
        ...
      spec:
        selector:
          app: istio-ingressgateway
          istio: ingressgateway
          revision: 1-18
        servers:
        - hosts:
          - api.example.com
          port:
            name: http-8080-api-example-com
            number: 8080
            protocol: HTTP
        
    3. Check the Envoy filter for the gateway.
        kubectl get envoyfilters -A
      kubectl get envoyfilter <istio-ingressgateway-pod-name> -n gloo-mesh-gateways -o yaml  
        
      In the output, verify the route configuration details, such as the HTTP route type, gateway context, route name, and Envoy filter details for the policy that you applied.
        ...
      spec:
        configPatches:
        - applyTo: HTTP_ROUTE
          match:
            context: GATEWAY
            routeConfiguration:
              vhost:
                route:
                  name: tracks-api-tracks-rt.gloo-mesh-gateways.cluster1-portal--api-example-com-rt.gloo-mesh-gateways.cluster1-portal
          patch:
            operation: MERGE
            value:
              route:
                rateLimits:
                - actions:
                  - genericKey:
                      descriptorValue: gloo-mesh.usage-plans-gloo-mesh-cluster1-portal-rate-limiter
                  - genericKey:
                      descriptorValue: solo.setDescriptor.uniqueValue
                  - requestHeaders:
                      descriptorKey: usagePlan
                      headerName: x-solo-plan
                      skipIfAbsent: true
                  - metadata:
                      descriptorKey: userId
                      metadataKey:
                        key: envoy.filters.http.ext_authz
                        path:
                        - key: userId
                  stage: 1
        
  5. Review the Envoy dashboard to check the filter order of the applied policies. For more information about filter order, see the Life of a Request in the Envoy docs.

    1. Get the name of the pods that run an Istio proxy, such as your Istio ingress gateway or a workload pod with an Istio-injected sidecar. The following example uses the default ingress gateway.

        kubectl get pods -n gloo-mesh-gateways
        
    2. Launch the Envoy dashboard for your gateway with the name of the pod that you previously retrieved.

        istioctl dashboard envoy -n gloo-mesh-gateways <istio-ingressgateway-pod-name>
        
    3. In your browser that opens to https://localhost:15000, click the config_dump command, such as shown in the following example. Optionally, you can enter dynamic_listeners in The resource to dump box to filter the output to the resource you want to review.

      Figure: Screenshot of envoy config dump interactive table of commands
      Figure: Screenshot of envoy config dump interactive table of commands
    4. In the config dump, search for dynamic_listeners for the 8080 HTTP port.

    5. In the listener’s filters_chain, review the order of the filters to find your policy. Typically, your policy’s filter is part of the HTTP Connection Manager (envoy.filters.network.http_connection_manager).

    6. If your policy is not in the expected order, you might need to check for multiple, conflicting policies or use a phase or stage setting in the policy to change its order (if available).

    7. If your policy is not in the filter chain, you might need to fix the policy configuration and re-apply the policy.

      Example filter chain output: In the following example, the rate limit filter happens after the external auth filter. The rate limit filter includes the configuration details that you set in the Gloo rate limit custom resources.

        "dynamic_listeners": [
       {
        "name": "0.0.0.0_8080",
        "active_state": {
         "version_info": "2023-04-12T16:51:47Z/9",
         "listener": {
          "@type": "type.googleapis.com/envoy.config.listener.v3.Listener",
          "name": "0.0.0.0_8080",
          "address": {
           "socket_address": {
            "address": "0.0.0.0",
            "port_value": 8080
           }
          },
          "filter_chains": [
           {
            "filters": [
             {
              "name": "istio_authn",
              "typed_config": {
               "@type": "type.googleapis.com/udpa.type.v1.TypedStruct",
               "type_url": "type.googleapis.com/io.istio.network.authn.Config"
              }
             },
             {
              "name": "envoy.filters.network.http_connection_manager",
              ...
              {
                "name": "envoy.filters.http.ext_authz",
                "typed_config": {
                 "@type": "type.googleapis.com/envoy.extensions.filters.http.ext_authz.v3.ExtAuthz",
                 "grpc_service": {
                  "envoy_grpc": {
                   "cluster_name": "outbound|8083||ext-auth-service.gloo-mesh.svc.cluster.local",
                   "authority": "outbound_.8083_._.ext-auth-service.gloo-mesh.svc.cluster.local"
                  },
                  "timeout": "2s"
                 },
                 "metadata_context_namespaces": [
                  "envoy.filters.http.jwt_authn"
                 ],
                 "transport_api_version": "V3"
                }
               },
               {
                "name": "envoy.filters.http.ratelimit",
                "typed_config": {
                 "@type": "type.googleapis.com/envoy.extensions.filters.http.ratelimit.v3.RateLimit",
                 "domain": "solo.io",
                 "stage": 1,
                 "request_type": "both",
                 "timeout": "0.100s",
                 "rate_limit_service": {
                  "grpc_service": {
                   "envoy_grpc": {
                    "cluster_name": "outbound|8083||rate-limiter.gloo-mesh.svc.cluster.local",
                    "authority": "outbound_.8083_._.rate-limiter.gloo-mesh.svc.cluster.local"
                   }
                  },
                  "transport_api_version": "V3"
                 }
                }
               },