After applying a policy, you might notice unexpected behavior or conversely no effect on your traffic.

Before you begin, you can review the Policy enforcement concepts to understand more about how policies are applied, import and export workspace considerations, and supported policy for type of gateway.

  1. Check your Gloo policy resources in the Gloo UI or your terminal.

  2. In your policy details, check for common issues, such as the following:

    • Any configuration error messages, including the following table of messages
    • Underlying servers that must exist, such as the rate limit or external auth server configurations for those policies
    • Correctly selected routes, destinations, or workloads
    • Correctly imported or exported resources
    • Typos in resource labels, names, namespaces
    MessageDescriptionSteps to Resolve
    conflicts with existing policy applied to route. Skipping applying <policy name>Some policies, such as JWT or header manipulation, cannot apply multiple policies to the same route at a time. If the policy selects several routes with the same label, the policy still applies to other routes that do not have a conflicting policy. The policy with the oldest creationTimestamp takes precedence.Revise the policies to select the route in only one policy. You can review which policies are applied to the route by checking the route status, such as in the Gloo UI.
  3. For policies that select imported and exported resources: In general, Gloo workspaces are set up to give you control of your microservices. When it comes to sharing policies across workspaces, policies generally still apply the rules of the source workspace to routes, destinations, or workspaces, even when those resources are shared across workspaces.

  4. Confirm that the policy selects the correct, healthy upstream service.

  5. Check the underlying Istio and Envoy resources that set up routing behavior in your Gloo environment. For Gloo Mesh Gateway, these resources are typically in the same namespace as your ingress gateway (such as gloo-mesh-gateways). For Gloo Mesh, these resources can be in all of the namespaces in your service mesh.

    1. Check the Istio virtual service for the host and route that you previously identified.
        kubectl get virtualservice -A
      kubectl get virtualservice <virtual-service-name> -n gloo-mesh-gateways -o yaml
        
      In the output, verify that the host, route, service, and gateway all match what you expect.
        
      gateways:
      - <virtual-gateway-name>
      hosts:
      - api.example.com
      http:
      - match:
        - sourceLabels:
            app: istio-ingressgateway
            istio: ingressgateway
            revision: 1-18
          uri:
            prefix: /trackapi/
        name: tracks-api-tracks-rt.gloo-mesh-gateways.cluster1-portal--api-example-com-rt.gloo-mesh-gateways.cluster1-portal
        rewrite:
          uri: /
        route:
        - destination:
            host: tracks-rest-api.tracks.svc.cluster.local
            port:
              number: 5000
        
    2. Check the matching Istio gateway.
        kubectl get gateways -A
      kubectl get gateway <virtual-gateway-name> -n gloo-mesh-gateways -o yaml
        
      In the output, confirm that the gateway, host, and protocol all match what you expect.
        ...
      spec:
        selector:
          app: istio-ingressgateway
          istio: ingressgateway
          revision: 1-18
        servers:
        - hosts:
          - api.example.com
          port:
            name: http-8080-api-example-com
            number: 8080
            protocol: HTTP
        
    3. Check the Envoy filter for the gateway.
        kubectl get envoyfilters -A
      kubectl get envoyfilter <istio-ingressgateway-pod-name> -n gloo-mesh-gateways -o yaml  
        
      In the output, verify the route configuration details, such as the HTTP route type, gateway context, route name, and Envoy filter details for the policy that you applied.
        ...
      spec:
        configPatches:
        - applyTo: HTTP_ROUTE
          match:
            context: GATEWAY
            routeConfiguration:
              vhost:
                route:
                  name: tracks-api-tracks-rt.gloo-mesh-gateways.cluster1-portal--api-example-com-rt.gloo-mesh-gateways.cluster1-portal
          patch:
            operation: MERGE
            value:
              route:
                rateLimits:
                - actions:
                  - genericKey:
                      descriptorValue: gloo-mesh.usage-plans-gloo-mesh-cluster1-portal-rate-limiter
                  - genericKey:
                      descriptorValue: solo.setDescriptor.uniqueValue
                  - requestHeaders:
                      descriptorKey: usagePlan
                      headerName: x-solo-plan
                      skipIfAbsent: true
                  - metadata:
                      descriptorKey: userId
                      metadataKey:
                        key: envoy.filters.http.ext_authz
                        path:
                        - key: userId
                  stage: 1
        
  6. Review the Envoy dashboard to check the filter order of the applied policies. For more information about filter order, see the Life of a Request in the Envoy docs.

    1. Get the name of the pods that run an Istio proxy, such as your Istio ingress gateway or a workload pod with an Istio-injected sidecar. The following example uses the default ingress gateway.

        kubectl get pods -n gloo-mesh-gateways
        
    2. Launch the Envoy dashboard for your gateway with the name of the pod that you previously retrieved.

        istioctl dashboard envoy -n gloo-mesh-gateways <istio-ingressgateway-pod-name>
        
    3. In your browser that opens to https://localhost:15000, click the config_dump command, such as shown in the following example. Optionally, you can enter dynamic_listeners in The resource to dump box to filter the output to the resource you want to review.

      Figure: Screenshot of envoy config dump interactive table of commands
      Figure: Screenshot of envoy config dump interactive table of commands
    4. In the config dump, search for dynamic_listeners for the 8080 HTTP port.

    5. In the listener’s filters_chain, review the order of the filters to find your policy. Typically, your policy’s filter is part of the HTTP Connection Manager (envoy.filters.network.http_connection_manager).

    6. If your policy is not in the expected order, you might need to check for multiple, conflicting policies or use a phase or stage setting in the policy to change its order (if available).

    7. If your policy is not in the filter chain, you might need to fix the policy configuration and re-apply the policy.

      Example filter chain output: In the following example, the rate limit filter happens after the external auth filter. The rate limit filter includes the configuration details that you set in the Gloo rate limit custom resources.

        "dynamic_listeners": [
       {
        "name": "0.0.0.0_8080",
        "active_state": {
         "version_info": "2023-04-12T16:51:47Z/9",
         "listener": {
          "@type": "type.googleapis.com/envoy.config.listener.v3.Listener",
          "name": "0.0.0.0_8080",
          "address": {
           "socket_address": {
            "address": "0.0.0.0",
            "port_value": 8080
           }
          },
          "filter_chains": [
           {
            "filters": [
             {
              "name": "istio_authn",
              "typed_config": {
               "@type": "type.googleapis.com/udpa.type.v1.TypedStruct",
               "type_url": "type.googleapis.com/io.istio.network.authn.Config"
              }
             },
             {
              "name": "envoy.filters.network.http_connection_manager",
              ...
              {
                "name": "envoy.filters.http.ext_authz",
                "typed_config": {
                 "@type": "type.googleapis.com/envoy.extensions.filters.http.ext_authz.v3.ExtAuthz",
                 "grpc_service": {
                  "envoy_grpc": {
                   "cluster_name": "outbound|8083||ext-auth-service.gloo-mesh.svc.cluster.local",
                   "authority": "outbound_.8083_._.ext-auth-service.gloo-mesh.svc.cluster.local"
                  },
                  "timeout": "2s"
                 },
                 "metadata_context_namespaces": [
                  "envoy.filters.http.jwt_authn"
                 ],
                 "transport_api_version": "V3"
                }
               },
               {
                "name": "envoy.filters.http.ratelimit",
                "typed_config": {
                 "@type": "type.googleapis.com/envoy.extensions.filters.http.ratelimit.v3.RateLimit",
                 "domain": "solo.io",
                 "stage": 1,
                 "request_type": "both",
                 "timeout": "0.100s",
                 "rate_limit_service": {
                  "grpc_service": {
                   "envoy_grpc": {
                    "cluster_name": "outbound|8083||rate-limiter.gloo-mesh.svc.cluster.local",
                    "authority": "outbound_.8083_._.rate-limiter.gloo-mesh.svc.cluster.local"
                   }
                  },
                  "transport_api_version": "V3"
                 }
                }
               },