Skip to content
home
main
Istio ambient mesh
Resiliency
Load balancing and failover
You are viewing the documentation for Solo Enterprise for Istio, formerly known as Gloo Mesh (OSS APIs). This version of the documentation is currently under development. Select latest from the version drop down or go to the landing page of the latest stable version.

L7 load balancing with kgateway

Page as Markdown
Configure load balancing and failover for traffic entering the ambient mesh through Solo Enterprise for kgateway.
About this guide

Traffic entering your ambient mesh from external clients can be load balanced and failed over using Solo Enterprise for kgateway as the ingress gateway. This guide shows how to configure Solo Enterprise for kgateway to route traffic to services in your ambient mesh, including routing through a waypoint proxy for L7 policy enforcement.
Solo Enterprise for kgateway does not originate HBONE connections. Instead, it uses a ztunnel to capture mesh traffic, behaving like a normal client within the mesh. This means that load balancing and failover policies are deferred to the ambient mesh components (ztunnel and waypoint) rather than being applied at the ingress gateway level.
For conceptual information about how load balancing works with ingress gateways, see the load balancing and failover overview.
Before you begin

Set up an ambient mesh in one cluster by using the Gloo Operator or Helm.
Step 1: Set up Solo Enterprise for kgateway

Install Solo Enterprise for kgateway in your cluster, with the Istio integration enabled. For more information, see the Solo Enterprise for kgateway setup guide and Istio integration guide.
  1. Set your Solo Enterprise for kgateway license key as an environment variable. If you do not have one, contact an account representative.
    export LICENSE_KEY=<license-key> 
  2. Deploy the Kubernetes Gateway API CRDs.
    kubectl apply -f https://github.com/kubernetes-sigs/gateway-api/releases/download/v1.5.0/standard-install.yaml
  3. Save the following Istio integration settings in an enterprise-kgateway.yaml Helm values file.
    cat <<EOF > enterprise-kgateway.yaml
controller:
  extraEnv:
    KGW_ENABLE_ISTIO_INTEGRATION: true
EOF
  4. Deploy the Solo Enterprise for kgateway CRDs by using Helm.
    helm upgrade -i enterprise-kgateway-crds oci://us-docker.pkg.dev/solo-public/enterprise-kgateway/charts/enterprise-kgateway-crds \
  --create-namespace \
  --namespace kgateway-system \
  --version 2.2.0-rc.2
  5. Install Solo Enterprise for kgateway, including your custom values file, by using Helm.
    helm upgrade -i enterprise-kgateway oci://us-docker.pkg.dev/solo-public/enterprise-kgateway/charts/enterprise-kgateway \
  -n kgateway-system \
  --version 2.2.0-rc.2 \
  --set-string licensing.licenseKey=${LICENSE_KEY} \
  -f enterprise-kgateway.yaml
  6. Make sure that the enterprise-kgateway control plane is running.
    kubectl get pods -n kgateway-system
    Example output:
    NAME                                   READY   STATUS    RESTARTS   AGE
enterprise-kgateway-5495d98459-46dpk   1/1     Running   0          19s

  7. Add the kgateway-system namespace to your ambient mesh. The label instructs istiod to configure a ztunnel socket on all the pods in that namespace so that traffic from these pods is secured via mutual TLS (mTLS).
    kubectl label ns kgateway-system istio.io/dataplane-mode=ambient
  8. Create a Gateway resource in the kgateway-system namespace with an HTTP listener.
    kubectl apply -f- <<EOF
apiVersion: gateway.networking.k8s.io/v1
kind: Gateway
metadata:
  name: http
  namespace: kgateway-system
spec:
  gatewayClassName: enterprise-kgateway
  listeners:
  - name: http
    port: 8080
    protocol: HTTP
    allowedRoutes:
      namespaces:
        from: All
EOF
  9. Wait for the Gateway to be ready.
    kubectl get gateway http -n kgateway-system
    Example output:
    NAME   CLASS                 ADDRESS        PROGRAMMED   AGE
http   enterprise-kgateway   10.96.123.45   True         30s
  10. Get the external address of the gateway and save it in an environment variable.
    export KGATEWAY_IP=$(kubectl get svc -n kgateway-system http -o=jsonpath="{.status.loadBalancer.ingress[0]['hostname','ip']}")
echo $KGATEWAY_IP
    kubectl port-forward -n kgateway-system deployment/http 8080:8080 &
export KGATEWAY_IP=localhost
Step 2: Deploy a sample app and test ingress routing

Deploy a backend app and configure an HTTPRoute to route traffic from the ingress gateway.
Traffic flow without waypoint: The following diagram shows how external traffic flows through the ingress gateway to reach backend services in the ambient mesh. Solo Enterprise for kgateway uses a ztunnel to enter the mesh, behaving like a normal mesh client. The ztunnel handles L4 load balancing.
    graph LR
    External[External client] -->|HTTP| KGW[Solo Enterprise<br/>for kgateway]
    KGW -->|mTLS mesh<br/>traffic| Ztunnel[kgateway ztunnel<br/>L4 load balancing]
    Ztunnel -->|Round-robin| Backend1[in-ambient pod 1]
    Ztunnel -->|Round-robin| Backend2[in-ambient pod 2]
    Ztunnel -->|Round-robin| Backend3[in-ambient pod 3]

    style KGW fill:#2068F3,color:#fff
  
  1. Deploy the in-ambient httpbin sample app. This manifest creates the httpbin namespace with an in-ambient backend service. The app is already labeled for inclusion in the ambient mesh with istio.io/dataplane-mode: ambient.
    kubectl apply -f https://raw.githubusercontent.com/solo-io/doc-examples/main/istio/sample-apps/in-ambient.yaml
  2. Scale the in-ambient deployment to 3 replicas so that you can observe load balancing across multiple endpoints.
    kubectl scale deployment in-ambient -n httpbin --replicas=3
  3. Verify that all pods are running.
    kubectl get pods -n httpbin
    Example output, in which in-ambient runs as 3 replicas:
    NAME                                 READY   STATUS    RESTARTS   AGE
in-ambient-7d8f9b6c54-abc12          1/1     Running   0          45s
in-ambient-7d8f9b6c54-def34          1/1     Running   0          20s
in-ambient-7d8f9b6c54-ghi56          1/1     Running   0          20s
  4. Create an HTTPRoute to route ingress traffic from the ingress gateway to the in-ambient service.
    kubectl apply -f- <<EOF
apiVersion: gateway.networking.k8s.io/v1
kind: HTTPRoute
metadata:
  name: in-ambient
  namespace: httpbin
spec:
  parentRefs:
  - name: http
    namespace: kgateway-system
  hostnames:
  - "httpbin.example.com"
  rules:
  - matches:
    - path:
        type: PathPrefix
        value: /
    backendRefs:
    - name: in-ambient
      port: 8000
EOF
  5. Verify that the HTTPRoute is accepted.
    kubectl get httproute -n httpbin
    Example output:
    NAME         HOSTNAMES                  AGE
in-ambient   ["httpbin.example.com"]    30s
  6. Send requests to the in-ambient app through the gateway. The /hostname endpoint returns the pod hostname, showing which replica handled each request. Traffic follows the path of kgateway -> ztunnel -> backend. The ztunnel uses round robin L4 load balancing across the backend endpoints. Verify that you see responses from all 3 replicas.
    for i in $(seq 1 9); do
  curl -s -H "Host: httpbin.example.com" http://$KGATEWAY_IP/hostname
done
    Example output:
    in-ambient-7d8f9b6c54-abc12
in-ambient-7d8f9b6c54-def34
in-ambient-7d8f9b6c54-ghi56
in-ambient-7d8f9b6c54-abc12
...
  7. Verify that traffic between the gateway proxy and the in-ambient app is secured via mutual TLS. Because traffic in an ambient mesh is intercepted by the ztunnels that are co-located on the same node as the sending and receiving service, you can check the logs of the ztunnels.
    1. Find the NODE that one of the in-ambient backend pods runs on.
      kubectl get pods -n httpbin -o wide
      Example output:
      NAME                          READY   STATUS    RESTARTS   AGE   IP           NODE                                                  NOMINATED NODE   READINESS GATES
in-ambient-7d8f9b6c54-abc12   1/1     Running   0          22h   10.XX.X.XX   gke-ambient-default-pool-bb9a8da5-bdf4   <none>           <none>
    2. Find the ztunnel that runs on the same node as the in-ambient backend pod.
      kubectl get pods -n istio-system -o wide | grep ztunnel
    3. Check the logs of that ztunnel instance and verify that the source and destination workloads have a SPIFFE ID.
      kubectl logs -n istio-system <ztunnel-instance>
      Example output:
      2025-03-19T17:32:42.762545Z	info	http access	request complete	src.addr=10.0.71.117:42468 src.workload="http-9db6c8995-l54dw" src.namespace="kgateway-system" src.identity="spiffe://cluster.local/ns/kgateway-system/sa/http" dst.addr=10.0.65.144:15008 dst.hbone_addr=10.0.65.144:8080 dst.service="in-ambient.httpbin.svc.cluster.local" dst.workload="in-ambient-7d8f9b6c54-abc12" dst.namespace="httpbin" dst.identity="spiffe://cluster.local/ns/httpbin/sa/in-ambient" direction="inbound" method=GET path="/headers" protocol=HTTP1 response_code=200 host="www.example.com:8080" user_agent="curl/8.7.1" request_id="4c5fc679-c5cd-4721-8735-51bcdbea6e0f" duration="0ms"
2025-03-19T17:32:46.810472Z	info	access	connection complete	src.addr=10.0.71.117:42468 src.workload="http-9db6c8995-l54dw" src.namespace="kgateway-system" src.identity="spiffe://cluster.local/ns/kgateway-system/sa/http" dst.addr=10.0.65.144:15008 dst.hbone_addr=10.0.65.144:8080 dst.service="in-ambient.httpbin.svc.cluster.local" dst.workload="in-ambient-7d8f9b6c54-abc12" dst.namespace="httpbin" dst.identity="spiffe://cluster.local/ns/httpbin/sa/in-ambient" direction="inbound" bytes_sent=1290 bytes_recv=550 duration="6742ms"
Step 3: Route ingress traffic through a waypoint

To apply L7 policies such as DestinationRules for HTTP-aware failover, configure ingress traffic to flow through a waypoint proxy.
Traffic flow with waypoint: The following diagram shows how traffic flows when the istio.io/ingress-use-waypoint=true label is set on the namespace or service. The ingress gateway ztunnel routes to the waypoint. The waypoint then enforces L7 policies and routes to backend endpoints.
    graph LR
    External[External client] -->|HTTP| KGW[Solo Enterprise<br/>for kgateway]
    KGW -->|mTLS mesh<br/>traffic| Ztunnel[kgateway ztunnel<br/>Routes to waypoint]
    Ztunnel -->|HBONE| Waypoint[Waypoint proxy<br/>L7 policy enforcement]
    Waypoint -->|L7 load balancing| Backend1[in-ambient pod 1]
    Waypoint -->|L7 load balancing| Backend2[in-ambient pod 2]
    Waypoint -->|L7 load balancing| Backend3[in-ambient pod 3]

    style KGW fill:#2068F3,color:#fff
    style Waypoint fill:#2068F3,color:#fff
  
  1. Create a waypoint Gateway in the httpbin namespace. The waypoint enforces L7 policies for services in the namespace.
    kubectl apply -f- <<EOF
apiVersion: gateway.networking.k8s.io/v1
kind: Gateway
metadata:
  name: httpbin-waypoint
  namespace: httpbin
spec:
  gatewayClassName: istio-waypoint
  listeners:
  - name: mesh
    port: 15008
    protocol: HBONE
    allowedRoutes:
      namespaces:
        from: Same
EOF
  2. Wait for the waypoint to be fully deployed.
    kubectl -n httpbin rollout status deployment/httpbin-waypoint
    Example output:
    deployment "httpbin-waypoint" successfully rolled out
  3. Add the istio.io/ingress-use-waypoint=true label to the in-ambient service so that ingress traffic is routed through the waypoint.
    kubectl label service in-ambient -n httpbin \
  istio.io/ingress-use-waypoint=true
  4. Send a request through the gateway to verify that traffic flows through the waypoint.
    curl -s -H "Host: httpbin.example.com" http://$KGATEWAY_IP/hostname
  5. Review the waypoint logs to confirm traffic is being processed.
    kubectl logs -n httpbin deploy/httpbin-waypoint | tail -10
    Example output:
    [2025-03-06T17:15:23.456Z] "GET /hostname HTTP/1.1" 200 - via_upstream - "-" 0 32 5 4 "-"
"curl/7.88.1" "abc123-def456" "in-ambient.httpbin.svc.cluster.local:8000"
"10.10.0.15:8080" inbound-vip|8000|http|in-ambient.httpbin.svc.cluster.local
10.10.0.14:45678 10.96.45.123:8000 10.10.0.14:45678 - default
  6. Send multiple requests to verify round-robin load balancing at the waypoint.
    for i in $(seq 1 9); do
  curl -s -H "Host: httpbin.example.com" http://$KGATEWAY_IP/hostname
done
    Example output showing traffic distributed across all 3 replicas:
    in-ambient-7d8f9b6c54-abc12
in-ambient-7d8f9b6c54-def34
in-ambient-7d8f9b6c54-ghi56
in-ambient-7d8f9b6c54-abc12
...
Step 4: Test ingress failover

Test how failover works when backend endpoints become unavailable.
  1. Apply an Istio DestinationRule at the waypoint for failover control. The DestinationRule is enforced at the waypoint, not at the ingress gateway, which allows for HTTP-aware failover with outlier detection. The outlier detection settings eject an endpoint for 30 seconds after 5 consecutive 5xx errors, with detection checks running every 10 seconds.
    kubectl apply -f- <<EOF
apiVersion: networking.istio.io/v1
kind: DestinationRule
metadata:
  name: in-ambient-failover
  namespace: httpbin
spec:
  host: in-ambient
  trafficPolicy:
    loadBalancer:
      simple: ROUND_ROBIN
    outlierDetection:
      consecutive5xxErrors: 5
      interval: 10s
      baseEjectionTime: 30s
EOF
  2. Delete one of the in-ambient backend pods to simulate an endpoint failure.
    kubectl delete pod -n httpbin -l app=in-ambient --field-selector=status.phase=Running --wait=false | head -1
  3. Send requests through the gateway. Traffic is distributed among the remaining healthy endpoints. The waypoint applies the outlier detection rules from the DestinationRule.
    for i in $(seq 1 6); do
  curl -s --max-time 2 -H "Host: httpbin.example.com" http://$KGATEWAY_IP/hostname || echo 'request failed'
done
    Example output showing traffic distributed only among the 2 remaining healthy replicas:
    in-ambient-7d8f9b6c54-def34
in-ambient-7d8f9b6c54-ghi56
in-ambient-7d8f9b6c54-def34
in-ambient-7d8f9b6c54-ghi56
in-ambient-7d8f9b6c54-def34
in-ambient-7d8f9b6c54-ghi56
  4. Check the pod status immediately after deletion to see one replica unavailable.
    kubectl get pods -n httpbin -l app=in-ambient
    Example output showing one pod terminating or in ContainerCreating state:
    NAME                          READY   STATUS              RESTARTS   AGE
in-ambient-7d8f9b6c54-abc12   1/1     Terminating         0          5m
in-ambient-7d8f9b6c54-def34   1/1     Running             0          5m
in-ambient-7d8f9b6c54-ghi56   1/1     Running             0          5m
in-ambient-7d8f9b6c54-jkl78   0/1     ContainerCreating   0          2s
  5. Wait for the replacement pod to start and verify that all three replicas are running again.
    sleep 10
kubectl get pods -n httpbin -l app=in-ambient
    Example output showing all replicas healthy:
    NAME                          READY   STATUS    RESTARTS   AGE
in-ambient-7d8f9b6c54-def34   1/1     Running   0          5m
in-ambient-7d8f9b6c54-ghi56   1/1     Running   0          5m
in-ambient-7d8f9b6c54-jkl78   1/1     Running   0          12s
Cleanup

  1. You can optionally remove the resources that you created in this guide. If you want to continue to the other load balancing and failover guides, you can keep the httpbin namespace and apps for use in those guides as well.
    kubectl delete destinationrule in-ambient-failover -n httpbin
kubectl delete httproute in-ambient -n httpbin
kubectl delete gateway httpbin-waypoint -n httpbin
kubectl delete namespace httpbin
kubectl delete gateway http -n kgateway-system
  2. To uninstall Solo Enterprise for kgateway, see the uninstall guide.
Next steps

For multicluster failover scenarios, see Multicluster zone and region failover.
Waypoints (L7)Multicluster zone and region failover