Overview

Learn how load balancing, failover, and outlier detection work for services within an ambient mesh, as well as for global services in a multicluster ambient mesh.

About load balancing, failover, and outlier detection

Load balancing, failover, and outlier detection work together to distribute traffic and handle unhealthy endpoints in an ambient mesh. While related, each concept serves a distinct purpose:

Load balancing: How you distribute traffic across healthy endpoints, such as through traffic distribution modes.
Outlier detection: How you detect unhealthy endpoints, such as through error thresholds, health checks, and ejection time.
Failover: What you do when endpoints are detected as unhealthy, such as routing to a different locality or following a priority order.

How these mechanisms behave depends on where traffic is handled and how you configure them. Traffic can be handled at Layer 4 (L4) by the ztunnel or at Layer 7 (L7) by a waypoint proxy or ingress gateway. You configure behavior through Kubernetes service settings, Istio annotations, and optionally Istio DestinationRules, as described in the following sections.

Native traffic distribution in Kubernetes

By default, Kubernetes distributes traffic across all healthy endpoints using round-robin load balancing. Services in the ambient mesh use this base behavior. You can optionally override it with the following settings.

Traffic distribution: Kubernetes supports locality-aware traffic distribution via the spec.trafficDistribution field on the service. Set this to PreferClose (also known as PreferSameZone) to prefer endpoints in the same network, region, and zone as the client. This is currently the only mode that Kubernetes supports natively. For more granular control, use the Istio traffic distribution settings.
```
apiVersion: v1
kind: Service
metadata:
  name: my-service
spec:
  trafficDistribution: PreferClose
```
Strict locality only (no failover): You can set internalTrafficPolicy: Local on a Service to restrict traffic to endpoints on the same node as the requesting client. Only endpoints that match the routing preferences are used; if no endpoints match, traffic is dropped. Because this option disables failover, use it only when you intentionally want to avoid sending traffic to other endpoints.
```
apiVersion: v1
kind: Service
metadata:
  name: my-service
spec:
  internalTrafficPolicy: Local
```

Istio traffic distribution

Istio extends Kubernetes traffic distribution with additional modes for finer control over how endpoints are prioritized by locality. In addition to the Kubernetes-native PreferClose mode (also known as PreferSameZone), Istio provides modes such as PreferNetwork, PreferRegion, PreferSameNode, and Any (round robin with no locality preference).

Traffic distribution is typically applied to global services in multicluster setups. The actual load balancing and failover behavior varies based on the component that is involved in the request, such as ztunnel for L4 traffic, waypoints for L7 policy enforcement, or ingress gateways for external traffic.

Review the following sections to understand how to configure traffic distribution in your service mesh, how locality is determined for endpoints, the available traffic distribution modes, and further configuration you can achieve with Istio DestinationRules.

Configuration

You can set traffic distribution at the following levels, from highest to lowest precedence:

Service spec.trafficDistribution (Kubernetes-native, supports PreferClose only)

apiVersion: v1
kind: Service
metadata:
  name: my-service
spec:
  trafficDistribution: PreferClose

Service networking.istio.io/traffic-distribution annotation (Istio traffic distribution modes, extends Kubernetes-native modes)

apiVersion: v1
kind: Service
metadata:
  name: my-service
  annotations:
    networking.istio.io/traffic-distribution: PreferNetwork

ServiceEntry networking.istio.io/traffic-distribution annotation (Istio-extended, supports all modes). Use this when you use a ServiceEntry.

apiVersion: networking.istio.io/v1beta1
kind: ServiceEntry
metadata:
  name: my-service-entry
  annotations:
    networking.istio.io/traffic-distribution: PreferRegion

Default behavior when unspecified. This varies by component; see Component-specific behavior for per-component defaults.

Locality determination

Locality is the set of attributes that describe where an endpoint runs: region, zone, network, and optionally subzone and node. The mesh uses locality to decide which endpoints are closest to the client for load balancing and failover.

Ambient determines locality in the following ways:

Local endpoints: Istiod determines locality from workload and node topology. For guidance on setting locality labels on nodes, such as topology.kubernetes.io/region and topology.kubernetes.io/zone, see the Kubernetes topology and Istio locality documentation.
Remote endpoints (multicluster): For east-west gateway setups with load balancer services, the mesh uses labels on the remote peering gateway to determine the locality of an endpoint. For flat network setups, the mesh uses locality labels on the WorkloadEntries that are exchanged between clusters.

The client’s locality, such as the client’s ztunnel or the waypoint or ingress gateway, is compared to each backend endpoint’s locality. The traffic distribution setting on the service then decides how endpoints are prioritized. For example, the mesh might prefer the same zone first, then the same network, or round robin across all endpoints.

Best practice: Set locality labels on nodes and on gateways, or WorkloadEntries when applicable, so that traffic distribution behaves predictably.

Traffic distribution modes and endpoint priority

Traffic distribution modes define priority levels for endpoints, such as same zone before same region before cross-network. They drive both load balancing and failover.

PreferClose / PreferSameZone: Prefer endpoints in the same network, region, and zone.
PreferNetwork: Prefer endpoints in the same network only, such as in-cluster before cross-cluster.
PreferRegion: Prefer endpoints in the same network and region.
Any: No locality preference; round robin across all endpoints.
PreferSameNode: Prefer endpoints on the same node or subzone. This is the most granular mode and is available only via the Istio annotation.

Some modes can be set on the Kubernetes service, such as PreferClose / PreferSameZone. Others are available only via the Istio annotation. For the full list and where each can be set, see the following table. Note that when you do not set a mode, the load balancing and failover behavior differs by component. Refer to the component-specific behavior to see the default mode for each component.

Mode	What it considers	Priority levels	Can be set via service spec.trafficDistribution
`PreferClose` / `PreferSameZone`	Network, region, zone	P0: Same network + same region + same zone P1: Same network + same region (different zone) P2: Same network (different region/zone) P3: All cross-network endpoints (same lowest priority regardless of region/zone)	Yes
`PreferNetwork`	Network only	P0: Same network P1: All cross-network endpoints (same lowest priority regardless of network)	No (annotation only)
`Any`	None (round robin)	P0: All cross-network endpoints (same lowest priority, regardless of network, zone, region)	No (annotation only)
`PreferRegion`	Network, region	P0: Same network + same region P1: Same network (different region, e.g., achieved through VPC peering) P2: All cross-network endpoints (same lowest priority regardless of network or region)	No (annotation only)
`PreferSameNode`	Network, region, zone, subzone, node	P0: Same network + same region + same zone + same subzone + same node P1: Same network + same region + same zone + same subzone (different node) P2: Same network + same region + same zone (different node, subzone) P3: Same network + same region (different node, subzone, zone) P4: Same network (different node, subzone, zone, region) P5: All cross-network endpoints (same lowest priority regardless of network, region, node, subzone, zone)	No (annotation only)

When L7 components such as waypoints or Istio ingress gateways handle traffic under high load, Envoy might send some requests to lower-priority endpoints (farther localities) even when higher-priority endpoints are still healthy. This capacity-based behavior, called spillover, helps prevent overload and distribute load more evenly. This is distinct from traditional failover, which occurs when endpoints become unavailable. For details on how spillover works, see [L7 load-based spillover](#waypoint-spillover).

**Best practice**: Choose the coarsest mode that meets your failover and latency goals. For example, use PreferNetwork for multicluster, or PreferClose when you want zone or region awareness.

DestinationRules

Use a DestinationRule when traffic distribution configuration alone is not enough. For example, use one when you need an explicit failover order, HTTP-based outlier detection such as for consecutive 5xx, or custom distribution weights.

In the DestinationRule spec.trafficPolicy section you can configure:

outlierDetection: Conditions for ejecting unhealthy endpoints, such as consecutive 5xx errors.
loadBalancer.localityLbSetting: Options include failover, failoverPriority, and distribute. These patterns are mutually exclusive. For more information, see the Istio reference documentaion for OutlierDetection and LocalityLoadBalancerSetting.

DestinationRules are enforced at the waypoint, or at the Istio ingress gateway or sidecar, for the backend’s hostname. For example, for global services in a multicluster mesh setup, you apply the DestinationRule to the <svc>.<namespace>.mesh.internal global hostname. DestinationRules are not applied on the ztunnel path.

When a DestinationRule exists for a host, it overrides the traffic distribution setting for that host. The waypoint or gateway enforces the rule in both single cluster and multicluster meshes.

**Best practice**: Prefer Istio traffic distribution modes for simplicity. Add DestinationRules when you need HTTP health checks, ejection behavior, or custom failover priority.

DestinationRule examples

The following examples show how to configure DestinationRules for outlier detection and failover. These configurations apply to waypoints, Istio ingress gateways, kgateway, and sidecars.

DestinationRule for a specific service

This example shows how to configure failover and outlier detection for the Bookinfo productpage service after it is exposed globally with the solo.io/service-scope=global label. The host uses the global service hostname productpage.bookinfo.mesh.internal. The component (waypoint, gateway, or sidecar) routes to endpoints in the order specified by the failoverPriority labels. The outlier detection settings define when endpoints are ejected from the load balancing pool.

apiVersion: networking.istio.io/v1beta1
kind: DestinationRule
metadata:
  name: productpage-failover
  namespace: bookinfo
spec:
  host: productpage.bookinfo.mesh.internal
  trafficPolicy:
    loadBalancer:
      localityLbSetting:
        enabled: true
        failoverPriority:
        - topology.kubernetes.io/region
        - topology.kubernetes.io/zone
      simple: ROUND_ROBIN
    outlierDetection:
      consecutive5xxErrors: 5
      interval: 10s
      baseEjectionTime: 3m
      maxEjectionPercent: 50

DestinationRule for multiple services

This example uses a wildcard host to apply the same failover and outlier detection configuration to all global services in a namespace. The wildcard pattern matches all global service hostnames in the format *.namespace.mesh.internal.

apiVersion: networking.istio.io/v1beta1
kind: DestinationRule
metadata:
  name: bookinfo-namespace-failover
  namespace: bookinfo
spec:
  host: "*.bookinfo.mesh.internal"
  trafficPolicy:
    loadBalancer:
      localityLbSetting:
        enabled: true
        failoverPriority:
        - topology.kubernetes.io/region
    outlierDetection:
      consecutive5xxErrors: 5
      interval: 10s
      baseEjectionTime: 3m
      maxEjectionPercent: 50

Component-specific behavior

The following pages detail how load balancing, outlier detection, and failover work for each ambient mesh component:

ztunnel: L4 load balancing and failover for in-mesh traffic between services.
Waypoint: L7 load balancing and failover when using waypoint proxies for policy enforcement.
kgateway (Solo Enterprise for kgateway): Load balancing, failover, and outlier detection for external traffic entering the mesh through Solo Enterprise for kgateway.
Istio ingress gateway: Load balancing, failover, and outlier detection for external traffic entering the mesh through an Istio ingress gateway.
Sidecar: Load balancing, failover, and outlier detection when a client has a sidecar proxy.

Note that traffic distribution methods and failover behavior vary based on the components that are included in each type of request path.

Guides

Use the failover guides to set up and test load balancing, failover, and outlier detection in your ambient mesh:

Ztunnel (L4): Set up and test basic L4 load balancing and failover with ztunnel.
Waypoints (L7): Configure L7 load balancing and failover with waypoint proxies, including DestinationRules for HTTP-aware behavior.
L7 load balancing with Solo Enterprise for kgateway: Configure load balancing and failover for traffic entering the mesh through Solo Enterprise for kgateway.
Multicluster zone and region failover: Configure zone and region-aware failover for global services in a multicluster ambient mesh.

Component-specific behavior