Skip to content
You are viewing the latest documentation for Solo Enterprise for kgateway. To access the documentation for older versions, use the version switcher.

Ingress to multicluster mesh

Page as Markdown
Use Solo Enterprise for kgateway as the ingress gateway to a multicluster ambient mesh.
About ambient mesh

Solo collaborated with Google to develop ambient mesh, a new “sidecarless” architecture for the Istio service mesh. Ambient mesh uses node-level ztunnels to route and secure Layer 4 traffic between pods with mutual TLS (mTLS). Waypoint proxies enforce Layer 7 traffic policies whenever needed. To onboard apps into the ambient mesh, you simply label the namespace the app belongs to. Because no sidecars need to be injected in to your apps, ambient mesh significantly reduces the complexity of adopting a service mesh.
To learn more about ambient, see the ambient mesh documentation.
About this guide

This feature requires both a Solo Enterprise for Istio and a Solo Enterprise for kgateway license. If you do not have these licenses, contact an account representative. For more information, see the version compatability matrix.
Set up a multicluster ambient mesh and expose apps across multiple clusters with a global hostname. Then, use your gateway proxy to load balance ambient mesh traffic across your clusters.
This guide assumes that you have two clusters, ${REMOTE_CLUSTER1} and ${REMOTE_CLUSTER2}, that you want to install ambient meshes in and link together. Solo Enterprise for kgateway is installed in ${REMOTE_CLUSTER1} alongside your ambient mesh workloads. To try out the multicluster routing capabilities, you deploy the Bookinfo app in both clusters. Then, you expose the productpage app across clusters with a global hostname, productpage.bookinfo.mesh.internal. Solo Enterprise for kgateway uses the global hostname to route traffic to the productpage apps in both clusters.
Solo Enterprise for kgateway as an ingress gateway to a multicluster ambient mesh
Solo Enterprise for kgateway as an ingress gateway to a multicluster ambient mesh
Before you begin
 
  1. Follow the Get started guide to install Solo Enterprise for kgateway.
  2. Follow the Sample app guide to create a gateway proxy with an HTTP listener and deploy the httpbin sample app.
  3. Get the external address of the gateway and save it in an environment variable.
    export INGRESS_GW_ADDRESS=$(kubectl get svc -n kgateway-system http -o jsonpath="{.status.loadBalancer.ingress[0]['hostname','ip']}")
echo $INGRESS_GW_ADDRESS  
    kubectl port-forward deployment/http -n kgateway-system 8080:8080
Step 1: Enable the Istio integration

Upgrade your Solo Enterprise for kgateway installation to enable the Istio integration so that Solo Enterprise for kgateway works with Istio DestinationRules.
  1. Get the Helm values for your current Helm installation.
    helm get values enterprise-kgateway -n kgateway-system -o yaml > enterprise-kgateway.yaml
open enterprise-kgateway.yaml
  2. Add the following values to the Helm values file to enable the Istio integration in Solo Enterprise for kgateway.
    
controller:
  extraEnv:
    KGW_ENABLE_ISTIO_INTEGRATION: true
  3. Upgrade your Helm installation.
    helm upgrade -i --namespace kgateway-system --version 2.2.4 enterprise-kgateway oci://us-docker.pkg.dev/solo-public/enterprise-kgateway/charts/enterprise-kgateway -f enterprise-kgateway.yaml
Step 2: Set up a multicluster ambient mesh

  1. Follow the multicluster ambient mesh setup guide in the Solo Enterprise for Istio documentation to install ambient in two clusters, ${REMOTE_CLUSTER1} and ${REMOTE_CLUSTER2}. The steps include setting up a shared root of trust, installing ambient in each cluster, and linking both clusters to create your multicluster ambient mesh. You can choose between the following installation methods:
    This guide assumes that the gateway control plane is installed in ${REMOTE_CLUSTER1}.
  2. Install Bookinfo in your multicluster setup and add it to the ambient mesh.
    1. Expose the productpage app across both clusters with a global hostname.
  3. Add the kgateway-system namespace to your ambient mesh. This label ensures that traffic from the gateway proxy to your apps are secured via mTLS.
    kubectl label ns kgateway-system istio.io/dataplane-mode=ambient --context ${REMOTE_CONTEXT1}
  4. Before setting up routing through the ingress gateway, verify multicluster routing within the mesh.
    1. Make sure that you can route from the ratings app to the global hostname that the productpage apps are exposed on.
      kubectl -n bookinfo --context ${REMOTE_CONTEXT1} debug -i pods/$(kubectl get pod -l app=ratings \
--context ${REMOTE_CONTEXT1} -A -o jsonpath='{.items[0].metadata.name}') \
--image=curlimages/curl -- curl -vik http://productpage.bookinfo.mesh.internal:9080/productpage
    2. Scale down the productpage app in ${REMOTE_CLUSTER1}.
      kubectl scale deployment productpage-v1 -n bookinfo --context ${REMOTE_CONTEXT1} --replicas=0
    3. Repeat the request to the productpage app. Because the productpage app is scaled down in ${REMOTE_CLUSTER1}, traffic is forced to go to the productpage app in ${REMOTE_CLUSTER2}. Verify that you continue to see a 200 HTTP response code.
      kubectl -n bookinfo --context ${REMOTE_CONTEXT1} debug -i pods/$(kubectl get pod -l app=ratings \
--context ${REMOTE_CONTEXT1} -A -o jsonpath='{.items[0].metadata.name}') \
--image=curlimages/curl -- curl -vik http://productpage.bookinfo.mesh.internal:9080/productpage
    4. Scale up the productpage app in ${REMOTE_CLUSTER1}.
      kubectl scale deployment productpage-v1 -n bookinfo --context ${REMOTE_CONTEXT1} --replicas=1
  5. Create an HTTPRoute to expose the global hostname for the productpage app along the /productpage prefix path on the http Gateway that you created in the get started tutorial.
    Note that if your global service or service namespace has the solo.io/service-takeover label, this label does not impact ingress routing. The label is primarily used to redirect in-mesh east-west traffic that targets the original .svc.cluster.local hostname to the global hostname instead. To route ingress traffic to a global hostname, you must reference the global service in the backendRefs block of your HTTPRoute as shown in this example.
    kubectl apply --context ${REMOTE_CONTEXT1} -f- <<EOF                                                
apiVersion: gateway.networking.k8s.io/v1
kind: HTTPRoute
metadata:
  name: productpage
  namespace: kgateway-system
spec:            
  parentRefs: 
    - name: http          
      namespace: kgateway-system                    
  rules:          
    - matches:
      - path:
          type: PathPrefix
          value: /productpage
      backendRefs:
        - name: productpage.bookinfo.mesh.internal 
          port: 9080
          kind: Hostname
          group: networking.istio.io
EOF
    Hostname format: The hostname format in backendRefs depends on whether your mesh uses segments. Without segments, global services use the .mesh.internal suffix as shown above. If your mesh uses segments, the hostname uses the segment domain suffix instead (for example, productpage.bookinfo.<segment_domain>). For more information, see Service scope and segment domains.
    Port mapping: The port value in the backendRefs field must match the port that is defined on the auto-generated ServiceEntry for the global service, which equals the port field on the Kubernetes Service and not the pod’s container port. If your Service uses a named targetPort, such as targetPort: http, ztunnel resolves the named port to the correct container port automatically.
    Known issue: In Solo Enterprise for Istio versions before 1.29.2-patch0 (1.29.x) and 1.28.6 (1.28.x), named targetPort values are not correctly resolved, causing traffic to arrive at the wrong port. Upgrade to the fixed version or, as a workaround, use a numeric targetPort in your Kubernetes Service definition, such as targetPort: 10000.
  6. Verify multicluster routing through the ingress gateway.
    1. Send a request through the ingress gateway along the /productpage path. Verify that you get back a 200 HTTP response code.
      curl -I http://$INGRESS_GW_ADDRESS:8080/productpage     
      Example output:
      HTTP/1.1 200 OK
content-type: text/html; charset=utf-8
content-length: 5179
server: envoy
x-envoy-upstream-service-time: 133
    2. Scale down the productpage app in ${REMOTE_CLUSTER1}.
      kubectl scale deployment productpage-v1 -n bookinfo --context ${REMOTE_CONTEXT1} --replicas=0
    3. Repeat the request along the /productpage path. Because the product page app is scaled down in ${REMOTE_CLUSTER1}, traffic is forced to go to the productpage app in ${REMOTE_CLUSTER2}. Verify that you continue to see a 200 HTTP response code.
      curl -I http://$INGRESS_GW_ADDRESS:8080/productpage     
      Example output:
      HTTP/1.1 200 OK
content-type: text/html; charset=utf-8
content-length: 5179
server: envoy
x-envoy-upstream-service-time: 133
    4. Scale up the productpage app in ${REMOTE_CLUSTER1}.
      kubectl scale deployment productpage-v1 -n bookinfo --context ${REMOTE_CONTEXT1} --replicas=1
Global service considerations

Review considerations when using kgateway proxies to route ingress traffic to a globally exposed service.
Service scope and segment domains

When you expose a service across clusters with the solo.io/service-scope label, an auto-generated ServiceEntry is created with a global hostname. The format of that hostname is determined by whether your mesh uses segments:
Segments configuredHostname format
No<svc>.<namespace>.mesh.internal
Yes<svc>.<namespace>.<segment_domain>
Use the generated hostname as the name in the backendRefs block of your HTTPRoute:
backendRefs:
  - name: <svc>.<namespace>.<hostname_suffix>
    port: <port>
    kind: Hostname
    group: networking.istio.io
To verify which hostname was generated for your service, check the ServiceEntry resources in the istio-system namespace:
kubectl get serviceentry -n istio-system
How scope affects reachability from kgateway:
The solo.io/service-scope value does not change the hostname format or the HTTPRoute syntax, but it determines which kgateway proxy instances can reach the service:
  • solo.io/service-scope=global: The service is reachable from any kgateway instance across all peered clusters, regardless of segment.
  • solo.io/service-scope=segment: The service is only reachable from kgateway instances in the same segment. A kgateway proxy in a different segment cannot resolve the global hostname.
For more information, see the Solo Enterprise for Istio documentation:
Route ingress traffic through a waypoint

By default, the kgateway proxy sends traffic directly to the destination pod, bypassing any waypoint that might be configured for the destination. To route ingress traffic through a waypoint, add the istio.io/ingress-use-waypoint: "true" label to the destination service or its namespace. No configuration is required on the kgateway proxy. The destination’s ztunnel intercepts the traffic and routes it through the waypoint automatically.
Understand N/S vs E/W failover behavior

Failover to a peer cluster behaves differently depending on whether traffic enters through the kgateway ingress proxy (north-south) or originates from within the mesh (east-west).
For east-west traffic, the source workload’s ztunnel socket performs endpoint selection and respects Kubernetes pod readiness. When local pods are in a NotReady state, ztunnel excludes them from the endpoint set and routes to cross-cluster endpoints instead.
For north-south traffic through kgateway, the kgateway proxy resolves the global hostname to a single ServiceEntry VIP and does not have visibility into individual pod readiness behind that VIP. Cross-cluster failover triggers only when no local pod endpoints exist, such as when all local replicas are scaled to zero. Pods that are in a NotReady state do not trigger cross-cluster failover on the north-south path.
To configure outlier detection or DestinationRule-based failover for north-south traffic, see kgateway load balancing and failover.
Use kgateway with PeerAuthentication strict mode

The gateway proxy must be enrolled in the ambient mesh so that its outbound connections to backend services are captured by ztunnel and participate in mesh policy enforcement. A consequence of this enrollment is that ztunnel also captures inbound connections to the gateway pod. With a PeerAuthentication policy set to mtls.mode: STRICT, ztunnel requires mTLS on every connection it handles. Because external clients do not have mesh identities, their inbound connections are rejected with this error message:
connection closed due to policy rejection: explicitly denied by: istio-system/istio_converted_static_strict
The intended behavior for an ingress gateway is for inbound external traffic to bypass ztunnel entirely. The ambient.istio.io/bypass-inbound-capture: "true" annotation configures this explicitly, so that ztunnel captures only outbound connections from the gateway to backend services.
To allow inbound external connections while keeping outbound traffic in the mesh, add the ambient.istio.io/bypass-inbound-capture: "true" annotation to the kgateway pod in an EnterpriseKgatewayParameters resource.
kubectl apply -f - <<EOF
apiVersion: enterprisekgateway.solo.io/v1alpha1
kind: EnterpriseKgatewayParameters
metadata:
  name: <params-name>
  namespace: <gateway-namespace>
spec:
  kube:
    podTemplate:
      extraAnnotations:
        ambient.istio.io/bypass-inbound-capture: "true"
EOF
Your Gateway resource must reference the EnterpriseKgatewayParameters resource so that the annotation applies to the gateway pod. For an example of how to reference an EnterpriseKgatewayParameters resource from a Gateway, see Customize the gateway proxy.
Next

Now that you set up Solo Enterprise for kgateway as the ingress gateway for your multicluster ambient mesh, you can further control and secure ingress traffic with Policies.
Ingress to single cluster mesh
Was this page helpful?