Policies
Debug Gloo policies.
After applying a policy, you might notice unexpected behavior or conversely no effect on your traffic.
Before you begin, you can review the Policy enforcement concepts to understand more about how policies are applied, import and export workspace considerations, and supported policy for type of gateway.
Check your Gloo policy resources in the Gloo UI or your terminal.
In your policy details, check for common issues, such as the following:
- Any configuration error messages, including the following table of messages
- Underlying servers that must exist, such as the rate limit or external auth server configurations for those policies
- Correctly selected routes, destinations, or workloads
- Correctly imported or exported resources
- Typos in resource labels, names, namespaces
Message Description Steps to Resolve conflicts with existing policy applied to route. Skipping applying <policy name>
Some policies, such as JWT or header manipulation, cannot apply multiple policies to the same route at a time. If the policy selects several routes with the same label, the policy still applies to other routes that do not have a conflicting policy. The policy with the oldest creationTimestamp
takes precedence.Revise the policies to select the route in only one policy. You can review which policies are applied to the route by checking the route status, such as in the Gloo UI. For policies that select imported and exported resources: In general, Gloo workspaces are set up to give you control of your microservices. When it comes to sharing policies across workspaces, policies generally still apply the rules of the source workspace to routes, destinations, or workspaces, even when those resources are shared across workspaces.
Confirm that the policy selects the correct, healthy upstream service.
Check the underlying Istio and Envoy resources that set up routing behavior in your Gloo environment. For Gloo Mesh Gateway, these resources are typically in the same namespace as your ingress gateway (such as
gloo-mesh-gateways
). For Gloo Mesh, these resources can be in all of the namespaces in your service mesh.- Check the Istio virtual service for the host and route that you previously identified.In the output, verify that the host, route, service, and gateway all match what you expect.
kubectl get virtualservice -A kubectl get virtualservice <virtual-service-name> -n gloo-mesh-gateways -o yaml
gateways: - <virtual-gateway-name> hosts: - api.example.com http: - match: - sourceLabels: app: istio-ingressgateway istio: ingressgateway revision: 1-20 uri: prefix: /trackapi/ name: tracks-api-tracks-rt.gloo-mesh-gateways.cluster1-portal--api-example-com-rt.gloo-mesh-gateways.cluster1-portal rewrite: uri: / route: - destination: host: tracks-rest-api.tracks.svc.cluster.local port: number: 5000
- Check the matching Istio gateway.In the output, confirm that the gateway, host, and protocol all match what you expect.
kubectl get gateways -A kubectl get gateway <virtual-gateway-name> -n gloo-mesh-gateways -o yaml
... spec: selector: app: istio-ingressgateway istio: ingressgateway revision: 1-20 servers: - hosts: - api.example.com port: name: http-8080-api-example-com number: 8080 protocol: HTTP
- Check the Envoy filter for the gateway.In the output, verify the route configuration details, such as the HTTP route type, gateway context, route name, and Envoy filter details for the policy that you applied.
kubectl get envoyfilters -A kubectl get envoyfilter <istio-ingressgateway-pod-name> -n gloo-mesh-gateways -o yaml
... spec: configPatches: - applyTo: HTTP_ROUTE match: context: GATEWAY routeConfiguration: vhost: route: name: tracks-api-tracks-rt.gloo-mesh-gateways.cluster1-portal--api-example-com-rt.gloo-mesh-gateways.cluster1-portal patch: operation: MERGE value: route: rateLimits: - actions: - genericKey: descriptorValue: gloo-mesh.usage-plans-gloo-mesh-cluster1-portal-rate-limiter - genericKey: descriptorValue: solo.setDescriptor.uniqueValue - requestHeaders: descriptorKey: usagePlan headerName: x-solo-plan skipIfAbsent: true - metadata: descriptorKey: userId metadataKey: key: envoy.filters.http.ext_authz path: - key: userId stage: 1
- Check the Istio virtual service for the host and route that you previously identified.
Review the Envoy dashboard to check the filter order of the applied policies. For more information about filter order, see the Life of a Request in the Envoy docs.
Get the name of the pods that run an Istio proxy, such as your Istio ingress gateway or a workload pod with an Istio-injected sidecar. The following example uses the default ingress gateway.
kubectl get pods -n gloo-mesh-gateways
Launch the Envoy dashboard for your gateway with the name of the pod that you previously retrieved.
istioctl dashboard envoy -n gloo-mesh-gateways <istio-ingressgateway-pod-name>
In your browser that opens to
https://localhost:15000
, click the config_dump command, such as shown in the following example. Optionally, you can enterdynamic_listeners
in The resource to dump box to filter the output to the resource you want to review.In the config dump, search for
dynamic_listeners
for the 8080 HTTP port.In the listener’s
filters_chain
, review the order of the filters to find your policy. Typically, your policy’s filter is part of the HTTP Connection Manager (envoy.filters.network.http_connection_manager
).If your policy is not in the expected order, you might need to check for multiple, conflicting policies or use a
phase
orstage
setting in the policy to change its order (if available).If your policy is not in the filter chain, you might need to fix the policy configuration and re-apply the policy.
Example filter chain output: In the following example, the rate limit filter happens after the external auth filter. The rate limit filter includes the configuration details that you set in the Gloo rate limit custom resources.
"dynamic_listeners": [ { "name": "0.0.0.0_8080", "active_state": { "version_info": "2023-04-12T16:51:47Z/9", "listener": { "@type": "type.googleapis.com/envoy.config.listener.v3.Listener", "name": "0.0.0.0_8080", "address": { "socket_address": { "address": "0.0.0.0", "port_value": 8080 } }, "filter_chains": [ { "filters": [ { "name": "istio_authn", "typed_config": { "@type": "type.googleapis.com/udpa.type.v1.TypedStruct", "type_url": "type.googleapis.com/io.istio.network.authn.Config" } }, { "name": "envoy.filters.network.http_connection_manager", ... { "name": "envoy.filters.http.ext_authz", "typed_config": { "@type": "type.googleapis.com/envoy.extensions.filters.http.ext_authz.v3.ExtAuthz", "grpc_service": { "envoy_grpc": { "cluster_name": "outbound|8083||ext-auth-service.gloo-mesh.svc.cluster.local", "authority": "outbound_.8083_._.ext-auth-service.gloo-mesh.svc.cluster.local" }, "timeout": "2s" }, "metadata_context_namespaces": [ "envoy.filters.http.jwt_authn" ], "transport_api_version": "V3" } }, { "name": "envoy.filters.http.ratelimit", "typed_config": { "@type": "type.googleapis.com/envoy.extensions.filters.http.ratelimit.v3.RateLimit", "domain": "solo.io", "stage": 1, "request_type": "both", "timeout": "0.100s", "rate_limit_service": { "grpc_service": { "envoy_grpc": { "cluster_name": "outbound|8083||rate-limiter.gloo-mesh.svc.cluster.local", "authority": "outbound_.8083_._.rate-limiter.gloo-mesh.svc.cluster.local" } }, "transport_api_version": "V3" } } },