Canary deployments with Argo Rollouts

Use Argo Rollouts to manage canary deployment strategies for workloads in your Solo Enterprise for Istio ambient mesh, including support for global services and multicluster routing.

About

Argo Rollouts is a Kubernetes controller that manages pod lifecycles for advanced deployment strategies, such as blue-green and canary. When combined with Solo Enterprise for Istio’s global services feature, Argo Rollouts can drive a canary rollout while automatically managing the traffic split across local service FQDNs, global hostnames, ingress paths, and cross-cluster destinations.

This guide covers two traffic management approaches:

The Gateway API approach uses the Argo Rollouts Gateway API plugin to split traffic between stable and canary Kubernetes services via HTTPRoute backendRef weights. This is the recommended approach for ambient mesh environments.
The Istio VirtualService + DestinationRule subsets approach uses Argo Rollouts’ Istio API support to split traffic via destination rule subset labels and virtual service route weights. This approach works for both sidecar and ambient datapaths and is most relevant for sidecar-to-ambient migration scenarios.

Before you begin

Install a multicluster ambient mesh. The examples in this guide assume two clusters that run mesh workloads.

Save the kubeconfig contexts for each cluster.

export context1=<cluster1-context>
export context2=<cluster2-context>

Deploy a waypoint proxy in the default namespace on cluster1 and enroll the namespace to use it. Both traffic management approaches in this guide require a waypoint as the L7 enforcement point where the Argo-managed traffic split is applied.

kubectl --context ${context1} apply -f- <<EOF
apiVersion: gateway.networking.k8s.io/v1
kind: Gateway
metadata:
  name: waypoint
  namespace: default
spec:
  gatewayClassName: istio-waypoint
  listeners:
  - name: mesh
    port: 15008
    protocol: HBONE
EOF

kubectl --context ${context1} label namespace default istio.io/use-waypoint=waypoint

Install the kubectl argo rollouts plugin.

Create the argo-rollouts namespace and deploy the Argo Rollouts components on cluster1.

kubectl --context ${context1} create namespace argo-rollouts
kubectl --context ${context1} apply -n argo-rollouts \
  -f https://github.com/argoproj/argo-rollouts/releases/latest/download/install.yaml

Deploy a curl client on both clusters to use for traffic testing.

for ctx in ${context1} ${context2}; do
  kubectl --context ${ctx} apply -f- <<EOF
apiVersion: apps/v1
kind: Deployment
metadata:
  name: curl
  namespace: default
spec:
  replicas: 1
  selector:
    matchLabels:
      app: curl
  template:
    metadata:
      labels:
        app: curl
    spec:
      containers:
      - name: curl
        image: curlimages/curl:latest
        command: ["sleep", "infinity"]
EOF
done

Gateway API

Set up Argo Rollouts with Gateway API traffic management for a local service marked as global in Solo Enterprise for Istio. The walkthrough covers ingress, in-mesh, and cross-cluster traffic paths.

When you use Argo Rollouts with the Gateway API and Solo Enterprise for Istio’s global services feature, note that the Gateway API plugin is provider-generic and only works with *Route backendRefs that reference Kubernetes services. The plugin follows the paradigm that Argo Rollouts is built around Kubernetes services to manage stable and canary pods. If Argo Rollouts cannot discover the stable and canary services, the rollout does not execute. The Gateway API plugin associates the stable and canary services to the backendRefs in the specified *Route resource by name.

For a canary deployment strategy, the Gateway API plugin only manages *Route backendRefs that reference Kubernetes Service types. Other backendRef kinds, such as Hostname backendRefs for global service routing, cannot be used in the routes that Argo manages directly. To route ingress traffic to the global hostname alongside an Argo-managed traffic split, use a separate, unmanaged route as shown in Apply Gateway API resources.

Configure the Gateway API plugin

Configure the Gateway API traffic router plugin so that Argo Rollouts can manage HTTPRoute resources on the cluster.

Configure the Argo Rollouts Gateway API plugin.

kubectl --context ${context1} apply -f- <<EOF
apiVersion: v1
kind: ConfigMap
metadata:
  name: argo-rollouts-config
  namespace: argo-rollouts
data:
  trafficRouterPlugins: |-
    - name: "argoproj-labs/gatewayAPI"
      location: "https://github.com/argoproj-labs/rollouts-plugin-trafficrouter-gatewayapi/releases/latest/download/gateway-api-plugin-linux-amd64"
EOF

Grant Argo Rollouts permission to read and update HTTPRoute resources. The Gateway API plugin only modifies backendRef weights on existing HTTPRoutes — no other resource access is required.

kubectl --context ${context1} apply -f- <<EOF
apiVersion: rbac.authorization.k8s.io/v1
kind: ClusterRole
metadata:
  name: argo-rollouts-gateway-controller-role
  namespace: argo-rollouts
rules:
- apiGroups: ["gateway.networking.k8s.io"]
  resources: ["httproutes"]
  verbs: ["get", "list", "watch", "update", "patch"]
---
apiVersion: rbac.authorization.k8s.io/v1
kind: ClusterRoleBinding
metadata:
  name: argo-rollouts-gateway-admin
roleRef:
  apiGroup: rbac.authorization.k8s.io
  kind: ClusterRole
  name: argo-rollouts-gateway-controller-role
subjects:
- namespace: argo-rollouts
  kind: ServiceAccount
  name: argo-rollouts
EOF

Restart the Argo Rollouts pod to pick up the configuration changes.

kubectl --context ${context1} rollout restart deployment/argo-rollouts -n argo-rollouts

Create services

Apply the three Service resources needed for this approach:

helloworld-stable: Selects only the stable ReplicaSet pods. Argo updates the selector during a rollout.
helloworld-canary: Selects only the canary ReplicaSet pods. Argo updates the selector during a rollout.
helloworld: A global service that selects all pods (both stable and canary). This service is not managed by Argo. The solo.io/service-scope: global label registers the service with Solo Enterprise for Istio’s global services feature.

kubectl --context ${context1} apply -f- <<EOF
# stable svc — Argo updates selector to target only the stable RS pods
apiVersion: v1
kind: Service
metadata:
  labels:
    app: helloworld
  name: helloworld-stable
  namespace: default
spec:
  ports:
  - name: http
    port: 5000
    protocol: TCP
    targetPort: 5000
  selector:
    app: helloworld
  type: ClusterIP
---
# canary svc — Argo updates selector to target only the canary RS pods
apiVersion: v1
kind: Service
metadata:
  labels:
    app: helloworld
  name: helloworld-canary
  namespace: default
spec:
  ports:
  - name: http
    port: 5000
    protocol: TCP
    targetPort: 5000
  selector:
    app: helloworld
  type: ClusterIP
---
# global svc — selects both stable and canary RS pods; not managed by Argo
apiVersion: v1
kind: Service
metadata:
  labels:
    app: helloworld
    istio.io/ingress-use-waypoint: "true"
    solo.io/service-scope: global
  name: helloworld
  namespace: default
spec:
  ports:
  - name: http
    port: 5000
    protocol: TCP
    targetPort: 5000
  selector:
    app: helloworld
  type: ClusterIP
EOF

Apply Gateway API resources

Apply the HTTPRoute that configures the waypoint to split helloworld service traffic between the stable and canary services.

Solo Enterprise for Istio automatically extends an HTTPRoute attached to a global service to also apply to the corresponding global hostname. The helloworld-svc HTTPRoute therefore covers both the local service (helloworld.default.svc.cluster.local) and the global hostname (helloworld.default.mesh.internal) at the waypoint; no separate HTTPRoute for the global hostname is needed.

Apply the HTTPRoute that configures the waypoint to split helloworld service traffic between the stable and canary services at weights 100 and 0 respectively. Argo updates these weights as the rollout progresses.

kubectl --context ${context1} apply -f- <<EOF
# waypoint route — splits service traffic between stable and canary
# Argo manipulates the backendRef weights as the rollout progresses
apiVersion: gateway.networking.k8s.io/v1
kind: HTTPRoute
metadata:
  name: helloworld-svc
  namespace: default
spec:
  parentRefs:
  - group: ""
    kind: Service
    name: helloworld
  rules:
  - backendRefs:
    - group: ""
      kind: Service
      name: helloworld-stable
      port: 5000
      weight: 100
    - group: ""
      kind: Service
      name: helloworld-canary
      port: 5000
      weight: 0
EOF

If you also have an ingress gateway, apply the following resources to verify the traffic split on the ingress path.

kubectl --context ${context1} apply -f- <<EOF
# ingress gateway
apiVersion: gateway.networking.k8s.io/v1
kind: Gateway
metadata:
  name: helloworld
  namespace: default
spec:
  gatewayClassName: istio
  listeners:
  - allowedRoutes:
      namespaces:
        from: Same
    name: http
    port: 80
    protocol: HTTP
---
# ingress route — directs traffic to the global service hostname
apiVersion: gateway.networking.k8s.io/v1
kind: HTTPRoute
metadata:
  name: helloworld-gw
  namespace: default
spec:
  parentRefs:
  - group: gateway.networking.k8s.io
    kind: Gateway
    name: helloworld
  rules:
  - backendRefs:
    - group: "networking.istio.io"
      kind: Hostname
      name: "helloworld.default.mesh.internal"
      port: 5000
EOF

Create the rollout

Create the Rollout resource that ties together the stable and canary services and the helloworld-svc HTTPRoute, and defines the canary promotion steps Argo executes.

Apply the Rollout resource. Note the following before applying:

No helloworld instances are running yet. Argo creates and manages both the stable and canary ReplicaSets based on the template spec.
The selector.matchLabels and template.metadata.labels align with the selectors defined on the stable and canary services. This alignment is required by Argo.

kubectl --context ${context1} apply -f- <<EOF
apiVersion: argoproj.io/v1alpha1
kind: Rollout
metadata:
  name: helloworld
  namespace: default
  labels:
    app: helloworld
spec:
  replicas: 1
  revisionHistoryLimit: 2
  selector:
    matchLabels:
      app: helloworld
  template:
    metadata:
      labels:
        app: helloworld
    spec:
      containers:
      - name: helloworld
        image: docker.io/istio/examples-helloworld-v1:1.0
        ports:
        - containerPort: 5000
        resources:
          requests:
            cpu: "100m"
        imagePullPolicy: IfNotPresent
  strategy:
    canary:
      stableService: helloworld-stable
      canaryService: helloworld-canary
      trafficRouting:
        plugins:
          argoproj-labs/gatewayAPI:
            httpRoutes:
            - name: helloworld-svc
            namespace: default
      steps:
      - pause: {}
      - setWeight: 50
      - pause: {}
EOF

After applying, a new helloworld ReplicaSet is created. Both the stable and canary services are updated to include the rollouts-pod-template-hash selector pointing to this ReplicaSet. Because no upgrade has occurred yet, both services select the same (and only) ReplicaSet.

Check the state of the rollout.

kubectl argo rollouts get rollout helloworld --context ${context1} -n default

Example output:

Name:            helloworld
Namespace:       default
Status:          ✔ Healthy
Strategy:        Canary
  Step:          4/4
  SetWeight:     100
  ActualWeight:  100
Images:          docker.io/istio/examples-helloworld-v1:1.0 (stable)
Replicas:
  Desired:       1
  Current:       1
  Updated:       1
  Ready:         1
  Available:     1

NAME                                  KIND        STATUS     AGE  INFO
⟳ helloworld                          Rollout     ✔ Healthy  10m
└──# revision:1                                              10m
   └──⧉ helloworld-9f8c6b2d4a         ReplicaSet  ✔ Healthy  10m  stable
      └──□ helloworld-9f8c6b2d4a-xyz  Pod         ✔ Running  10m  ready:2/2

Run the canary rollout

Trigger a canary rollout, verify the traffic split, and promote through the rollout steps to complete the upgrade.

Trigger the canary deployment by updating the image in the pod template spec.
```
kubectl argo rollouts set image helloworld \
  helloworld=docker.io/istio/examples-helloworld-v2:1.0 \
  --context ${context1} -n default
```
Argo creates the canary ReplicaSet with the new image. Because the first step in the canary strategy is pause, the canary ReplicaSet starts with zero replicas. Both stable and canary service selectors still point to the stable ReplicaSet.

Wait for the rollout to show Paused status before promoting.

kubectl argo rollouts get rollout helloworld --context ${context1} -n default

Promote the rollout to move to the setWeight: 50 step.

kubectl argo rollouts promote helloworld --context ${context1} -n default

After promoting, verify that:

Both ReplicaSets have 1 replica each. The stable ReplicaSet remains running and the canary ReplicaSet scales up from 0 to 1. You can identify the canary ReplicaSet by the recency of its age, such as 2m in the following example output.
```
kubectl --context ${context1} get replicasets -n default -l app=helloworld
```
Example output:
```
NAME                    DESIRED   CURRENT   READY   AGE
helloworld-74d4db7bbc   1         1         1       2m
helloworld-9f8c6b2d4a   1         1         1       10m
```

The helloworld-canary Service selector now includes the rollouts-pod-template-hash of the canary ReplicaSet.

kubectl --context ${context1} get svc helloworld-canary -n default \
  -o jsonpath='{.spec.selector}'

Example output:

{"app":"helloworld","rollouts-pod-template-hash":"74d4db7bbc"}

The helloworld-svc HTTPRoute backendRef weights updated to 50/50.

kubectl --context ${context1} get httproute helloworld-svc -n default \
  -o jsonpath='{range .spec.rules[0].backendRefs[*]}{.name}{"\t"}{.weight}{"\n"}{end}'

Example output:

helloworld-stable  50
helloworld-canary  50

Verify the traffic split across the in-mesh paths.

# local, in-mesh path
for i in $(seq 1 50); do
  kubectl --context ${context1} -n default exec deploy/curl -- \
    curl -s helloworld.default.mesh.internal:5000/hello
done | sort | uniq -c | sort -rn

# remote, in-mesh path
for i in $(seq 1 50); do
  kubectl --context ${context2} -n default exec deploy/curl -- \
    curl -s helloworld.default.mesh.internal:5000/hello
done | sort | uniq -c | sort -rn

The output shows an approximately 50/50 split between v1 and v2.

If you applied the ingress gateway resources, also verify the ingress path.

INGRESS_GW_ADDRESS=$(kubectl --context ${context1} get svc -n default helloworld-istio \
  -o jsonpath='{.status.loadBalancer.ingress[0].ip}')
for i in $(seq 1 50); do
  curl -s --resolve helloworld.istio:80:$INGRESS_GW_ADDRESS http://helloworld.istio/hello
done | sort | uniq -c | sort -rn

The output shows an approximately 50/50 split between v1 and v2.

Promote the rollout to complete the canary strategy.
```
kubectl argo rollouts promote helloworld --context ${context1} -n default
```
After the final promotion:
- The canary ReplicaSet is promoted to become the new stable ReplicaSet.
- The previously stable ReplicaSet is scaled to zero.
- Both stable and canary service selectors point to the new stable ReplicaSet.
- The helloworld-svc HTTPRoute backendRef weights reset to 100 and 0.

You have successfully upgraded the application from docker.io/istio/examples-helloworld-v1:1.0 to docker.io/istio/examples-helloworld-v2:1.0 while managing a dynamic traffic split across the global service hostname for in-mesh and cross-cluster traffic paths.

Multicluster routing considerations

This example uses a simplistic multicluster configuration: helloworld is a global service with endpoints only in cluster1, and cluster2 has no waypoints. Routing behavior to the global service on cluster1 can change depending on cluster2’s configuration. The following scenarios on cluster2 can cause the Argo-managed traffic split to be bypassed.

Client-local waypoint: If cluster2’s configuration diverges from the example above, such as if the client namespace or the destination service on cluster2 is enrolled in a local waypoint, L7 policy is applied at cluster2’s waypoint instead. The request then routes through cluster1’s east-west gateway, bypassing cluster1’s waypoint where the Argo-managed HTTPRoute weights exist.
Local endpoints with failover routing: If cluster2 also marks its local helloworld service with solo.io/service-scope: global, cluster2’s helloworld endpoints join cluster1’s helloworld endpoints in the same helloworld.default.mesh.internal global service. Even without a waypoint on cluster2, the default routing behavior is failover. Traffic stays in-network and is served by cluster2’s local endpoints. Cluster1’s endpoints and the Argo-managed traffic split are only reached if cluster2’s local endpoints become unhealthy, or if you explicitly disable failover load balancing via a DestinationRule for that host.

Cleanup

Remove the resources created in this section.

kubectl --context ${context1} -n default delete rollout/helloworld
kubectl --context ${context1} -n default delete httproute/helloworld-svc
kubectl --context ${context1} -n default delete svc/helloworld-stable
kubectl --context ${context1} -n default delete svc/helloworld-canary
kubectl --context ${context1} -n default delete svc/helloworld
# If you applied the ingress gateway resources:
kubectl --context ${context1} -n default delete httproute/helloworld-gw
kubectl --context ${context1} -n default delete gateway/helloworld

VirtualService + DestinationRule subsets

Set up Argo Rollouts with Istio VirtualService and DestinationRule resources to achieve traffic management for a local service marked as global in Solo Enterprise for Istio, covering ingress, in-mesh, and cross-cluster traffic paths.

Considerations

Migration use case: The VirtualService + DestinationRule approach is the traditional sidecar path and is most relevant for sidecar-to-ambient migration scenarios, where sidecar and ambient-specific features might be temporarily mixed. However, this approach still works for both sidecar and ambient datapaths. In both cases, the VirtualService + DestinationRule are applied at the waypoint as the L7 enforcement point for the Argo-managed traffic split. Sidecar clients with ENABLE_WAYPOINT_INTEROP enabled bypass virtual service routing and connect directly to the waypoint via HBONE, where subset routing is applied. The traditional ingress gateway similarly forwards traffic to the waypoint via istio.io/ingress-use-waypoint.

ENABLE_WAYPOINT_INTEROP: This environment variable is a Solo Enterprise for Istio pilot environment variable (enabled by default in Solo builds) that allows sidecar proxies to short-circuit all processing and connect directly to a waypoint when the destination service has one. This is required for sidecar-to-ambient interoperability during migration.

Dataplane scope: The VirtualService + DestinationRule are applied only by Envoy-based dataplane components: sidecars, waypoints, and ingress/egress gateways. ztunnel (L4 only) and east-west HBONE gateways do not apply the VirtualService + DestinationRule. A waypoint must be present in the namespace for this approach to work in ambient mode.

Pure sidecar environments: If your environment already runs sidecar injection rather than ambient enrollment, the istio.io/dataplane-mode label is simply absent from your namespaces and the rest of this section applies as-is. The waypoint is the enforcement point in both cases.

Create a service

Apply a single Service resource for the application. Because Argo uses destination rule subsets to achieve the traffic split between stable and canary ReplicaSet pods, only one service is needed. The istio.io/ingress-use-waypoint label ensures that incoming traffic through the ingress gateway routes traffic through the waypoint, where the Argo-managed VirtualService + DestinationRule subset routing is applied.

kubectl --context ${context1} apply -f- <<EOF
apiVersion: v1
kind: Service
metadata:
  labels:
    app: helloworld
    istio.io/ingress-use-waypoint: "true"
    solo.io/service-scope: global
  name: helloworld
  namespace: default
spec:
  ports:
  - name: http
    port: 5000
    protocol: TCP
    targetPort: 5000
  selector:
    app: helloworld
  type: ClusterIP
EOF

Apply Istio traffic management resources

Apply the VirtualService and DestinationRule resources. The helloworld virtual service is bound to mesh and defines a primary route with stable and canary subset destinations. Solo Enterprise for Istio applies this virtual service at the waypoint for both the local service hostname and the global hostname. Because the virtual service hosts field controls which traffic matches and the route destination.host field controls where it goes, a single route using the local host covers both hostnames.

The DestinationRule defines stable and canary subsets for the local helloworld host. A single destination rule provides subset definitions for both traffic paths because all virtual service destinations reference the same local host.

Argo Rollouts manipulates the labels defined on the destination rule subsets and the helloworld virtual service destination weights to achieve dynamic traffic splitting during the canary strategy.

kubectl --context ${context1} apply -f- <<EOF
# Argo-managed virtual service — applied at the waypoint for mesh traffic
apiVersion: networking.istio.io/v1alpha3
kind: VirtualService
metadata:
  name: helloworld
  namespace: default
spec:
  gateways:
  - mesh
  hosts:
  - helloworld
  - helloworld.default.mesh.internal
  http:
  - name: primary
    route:
    - destination:
        host: helloworld
        subset: stable
      weight: 100
    - destination:
        host: helloworld
        subset: canary
      weight: 0
---
# Destination rule — defines subsets for the local helloworld host
apiVersion: networking.istio.io/v1alpha3
kind: DestinationRule
metadata:
  name: helloworld
  namespace: default
spec:
  host: helloworld
  subsets:
  - name: stable
    labels:
      app: helloworld
  - name: canary
    labels:
      app: helloworld
EOF

If you also have an ingress gateway, apply the following resources to verify the traffic split on the ingress path. The helloworld-ingress virtual service is bound to the Istio ingress Gateway and forwards traffic to the helloworld service without subsets. The istio.io/ingress-use-waypoint label on the service causes the ingress gateway to route through the waypoint, where the subset split is applied, instead of directly to the backends.

kubectl --context ${context1} apply -f- <<EOF
# Istio Gateway — configures the ingress gateway listener
apiVersion: networking.istio.io/v1alpha3
kind: Gateway
metadata:
  name: helloworld-ingress
  namespace: default
spec:
  selector:
    istio: ingressgateway
  servers:
  - hosts:
    - helloworld.istio
    port:
      name: http
      number: 80
      protocol: HTTP
---
# Ingress virtual service — forwards to helloworld service without subsets
# The waypoint handles the traffic split via the Argo-managed virtual service above
apiVersion: networking.istio.io/v1alpha3
kind: VirtualService
metadata:
  name: helloworld-ingress
  namespace: default
spec:
  gateways:
  - default/helloworld-ingress
  hosts:
  - helloworld.istio
  http:
  - name: ingress
    route:
    - destination:
        host: helloworld
      weight: 100
EOF

Create the rollout

Create the Rollout resource that ties together the helloworld service, VirtualService, and DestinationRule, and defines the canary promotion steps Argo executes.

Apply the Rollout resource. Note the following before applying:
- No helloworld instances are running yet. Argo creates and manages both the stable and canary ReplicaSets based on the template spec.
- The selector.matchLabels and template.metadata.labels align with the selectors defined on the helloworld service. This alignment is required by Argo.
- The helloworld virtual service is specified with the primary route so Argo updates its weights as the canary strategy progresses.
- The helloworld destination rule is specified with the stable and canary subset names so Argo updates their labels with the rollouts-pod-template-hash.
```
kubectl --context ${context1} apply -f- <<EOF
```

apiVersion: argoproj.io/v1alpha1 kind: Rollout metadata: name: helloworld namespace: default spec: replicas: 1 revisionHistoryLimit: 2 selector: matchLabels: app: helloworld template: metadata: labels: app: helloworld spec: containers: - name: helloworld image: docker.io/istio/examples-helloworld-v1:1.0 ports: - containerPort: 5000 resources: requests: cpu: “100m” imagePullPolicy: IfNotPresent strategy: canary: trafficRouting: istio: virtualService: name: helloworld routes: - primary destinationRule: name: helloworld canarySubsetName: canary stableSubsetName: stable steps: - pause: {} - setWeight: 50 - pause: {} EOF


After applying this resource, a new `helloworld` ReplicaSet is created. The destination rule's stable and canary subset labels are both updated to include the `rollouts-pod-template-hash` pointing to pods of this ReplicaSet. Because no upgrade has occurred yet, both subsets select the same (and only) ReplicaSet.

2. Check the state of the rollout.
```sh
kubectl argo rollouts get rollout helloworld --context ${context1} -n default

Example output:

Name:            helloworld
Namespace:       default
Status:          ✔ Healthy
Strategy:        Canary
  Step:          3/3
  SetWeight:     100
  ActualWeight:  100
Images:          docker.io/istio/examples-helloworld-v1:1.0 (stable)
Replicas:
  Desired:       1
  Current:       1
  Updated:       1
  Ready:         1
  Available:     1

NAME                                  KIND        STATUS     AGE  INFO
⟳ helloworld                          Rollout     ✔ Healthy  10m
└──# revision:1                                              10m
   └──⧉ helloworld-9f8c6b2d4a         ReplicaSet  ✔ Healthy  10m  stable
      └──□ helloworld-9f8c6b2d4a-xyz  Pod         ✔ Running  10m  ready:1/1

Run the canary rollout

Trigger a canary rollout, verify the traffic split, and promote through the rollout steps to complete the upgrade.

Trigger the canary deployment by updating the image in the pod template spec.
```
kubectl argo rollouts set image helloworld \
  helloworld=docker.io/istio/examples-helloworld-v2:1.0 \
  --context ${context1} -n default
```
Argo creates the canary ReplicaSet with the updated image. Because the first step in the canary strategy is pause, the canary ReplicaSet starts with zero replicas and the destination rule subset labels still point to the stable ReplicaSet.

Wait for the rollout to show Paused status before promoting.

kubectl argo rollouts get rollout helloworld --watch --context ${context1} -n default

Promote the rollout to move to the setWeight: 50 step.
```
kubectl argo rollouts promote helloworld --context ${context1} -n default
```
After promoting, verify the following:
- The canary ReplicaSet has 1 replica.
- The helloworld destination rule canary subset labels updated to include the rollouts-pod-template-hash selecting the new canary ReplicaSet pods.
- The helloworld virtual service primary route destination weights updated to 50/50.

Verify the traffic split across the in-mesh paths.

# local, in-mesh path
for i in $(seq 1 50); do
  kubectl --context ${context1} -n default exec deploy/curl -- \
    curl -s helloworld.default.mesh.internal:5000/hello
done | sort | uniq -c | sort -rn

# remote, in-mesh path
# works for both ambient and sidecar clients:
# ENABLE_WAYPOINT_INTEROP routes sidecar traffic through the waypoint,
# and ambient traffic is routed through the waypoint by default
for i in $(seq 1 50); do
  kubectl --context ${context2} -n default exec deploy/curl -- \
    curl -s helloworld.default.mesh.internal:5000/hello
done | sort | uniq -c | sort -rn

The output shows an approximately 50/50 split between v1 and v2.

If you applied the ingress gateway resources, also verify the ingress path.

INGRESS_GW_ADDRESS=$(kubectl --context ${context1} get svc istio-ingressgateway \
  -n istio-system -o jsonpath='{.status.loadBalancer.ingress[0].ip}')
for i in $(seq 1 50); do
  curl -s --resolve helloworld.istio:80:$INGRESS_GW_ADDRESS http://helloworld.istio/hello
done | sort | uniq -c | sort -rn

The output shows an approximately 50/50 split between v1 and v2.

Promote the rollout to complete the canary strategy.
```
kubectl argo rollouts promote helloworld --context ${context1} -n default
```
After the final promotion:
- The canary ReplicaSet is promoted to become the new stable ReplicaSet.
- The previously stable ReplicaSet is scaled to zero.
- The helloworld destination rule stable and canary subset labels both point to the new stable ReplicaSet’s rollouts-pod-template-hash.
- The helloworld virtual service primary route destination weights reset to 100 and 0.

Multicluster routing considerations

The example above uses a simplistic multicluster configuration: helloworld is a global service with endpoints only in cluster1. Routing behavior to the global service on cluster1 can change depending on cluster2’s client application and namespace configuration. The following scenarios on cluster2 cause the Argo-managed traffic split to be bypassed.

Client-local waypoint: L7 policy is applied at cluster2’s waypoint, bypassing the Argo-managed traffic split on cluster1.

Cluster2’s client namespace or the destination service is enrolled in a local waypoint, so L7 policy is enforced there.
The request routes through cluster1’s east-west gateway, bypassing cluster1’s waypoint where the Argo-managed policy lives.

Local endpoints with failover routing: Traffic stays in-network on cluster2 by default, bypassing cluster1’s endpoints and traffic split.

Cluster2 marks its local helloworld service with solo.io/service-scope: global, adding its endpoints to the same global federation.
The default behavior is failover: cluster2’s local endpoints serve traffic first.
To reach cluster1’s endpoints and the Argo-managed traffic split, either let cluster2’s local endpoints become unhealthy, or explicitly disable failover load balancing via a DestinationRule for that host.

Remote sidecar client without ENABLE_WAYPOINT_INTEROP: The remote sidecar applies L7 policy client-side and has no knowledge of the VirtualService + DestinationRule that Argo manages on cluster1.

Solo Enterprise for Istio does not peer VirtualServices or DestinationRules across clusters.
Enable ENABLE_WAYPOINT_INTEROP (the default in the Solo distribution of Istio) so the sidecar bypasses virtual service routing and connects to cluster1’s waypoint via the east-west gateway, where the Argo-managed traffic split is correctly applied.

Cross-cluster subset matching requires label propagation: Subset selectors do not match remote endpoints, causing cross-cluster requests via subsets to fail with 503: no_healthy_upstream.

The peering controller creates autogenerated WorkloadEntry objects in remote clusters with a minimal label set by default, so subset selectors (including the rollouts-pod-template-hash that Argo writes) do not match.
Set ENABLE_PEERING_LABEL_PROPAGATION on istiod to all or a comma-delimited list of label keys — at minimum app,rollouts-pod-template-hash — to propagate the labels needed for subset matching.

Cleanup

Remove the resources created in this section.

kubectl --context ${context1} -n default delete rollout/helloworld
kubectl --context ${context1} -n default delete dr/helloworld
kubectl --context ${context1} -n default delete vs/helloworld
kubectl --context ${context1} -n default delete svc/helloworld
# If you applied the ingress gateway resources:
kubectl --context ${context1} -n default delete vs/helloworld-ingress
kubectl --context ${context1} -n default delete gateway.networking.istio.io/helloworld-ingress

Flat network considerations

When two peered clusters share the same flat network, each pod workload is peered as an autogenflat.* WorkloadEntry in the remote cluster’s registry, and remote pod IPs are directly reachable without crossing an east-west gateway. This changes how traffic is routed at the L4 layer and opens additional patterns for the VirtualService + DestinationRule subset approach.

Traffic addressing and L7 policy enforcement

In a flat network, the L4 path to remote pods changes, but L7 policy enforcement at the waypoint stays the same as long as traffic is addressed via the service VIP or global hostname.

What changes compared to cross-network: ztunnel establishes HBONE connections directly to the remote cluster’s waypoint or directly to the pod when no waypoint is involved, skipping the east-west gateway.

What stays the same: Either approach to the Argo-managed L7 policy is enforced at the waypoint identically. Removing the east-west gateway from the L4 path does not change virtual service/HTTPRoute matching, subset selection, or weight application.

Addressing pods by IP bypasses the traffic split: WorkloadEntries that are auto-generated for flat-network remote pods have WAYPOINT: None in the remote cluster’s ztunnel registry, regardless of the waypoint enrollment on the source cluster’s namespace or service. When a client on cluster2 connects to a cluster1 pod IP directly, ztunnel routes to that pod’s local ztunnel via HBONE, bypassing the waypoint entirely. This is true even if the request carries a Host header matching the virtual service hosts field: ztunnel’s routing decision is IP-based, and the waypoint is never consulted. The Argo-managed traffic split does not apply.

To keep the Argo-managed traffic split working, remote clients on a flat network must address traffic via the service VIP or global hostname. Both cause ztunnel to route through the waypoint where the virtual service/HTTPRoute applies the traffic split. The flat-network benefit is L4 efficiency (no east-west gateway hop); the L7 abstraction still requires a service-level address.

Service VIP reachability across clusters depends on whether the service CIDR is routed between clusters at the network layer. In real infrastructure, the service CIDR is typically not advertised cross-cluster, and the global hostname is the portable choice.

Including remote pods in Argo-managed subsets (VirtualService + DestinationRule only)

Flat-network peering also opens the possibility of including remote pods in an Argo-managed traffic split. For example, you can have cluster2’s local helloworld pods participate in the stable and canary subsets of cluster1’s DestinationRule alongside cluster1’s own pods.

This consideration applies only to the VirtualService + DestinationRule subset approach. The Gateway API plugin requires Kubernetes service backendRefs, which inherently only select local workloads. Remote flat-network WorkloadEntries cannot be referenced as backendRefs.

Argo Rollouts writes the rollouts-pod-template-hash label into the destination rule’s stable and canary subset selectors. The hash value depends on the rollout’s state:

Steady state (no active canary): Both subsets select the current stable ReplicaSet’s hash.
Mid-rollout: The stable subset selects the old hash; the canary subset selects the new hash.
After rollout completes: Both subsets converge on the new stable hash.

For a remote cluster’s flat WorkloadEntry to be included in a subset, its propagated labels must match the subset’s selector–both app: helloworld and the exact rollouts-pod-template-hash value the subset currently requires.

For cluster2’s pods to participate in cluster1’s Argo-managed subsets, all of the following must be true.

Both clusters run Argo Rollouts managing the same application. A plain Kubernetes Deployment on cluster2 applies a different label (pod-template-hash, computed by the Deployment controller) that never matches cluster1’s subset selectors.
The pod template spec is identical across clusters. Argo’s hash computation is deterministic for a given template and collision count, in that identical templates produce identical hashes. Both inline pod templates Rollout.spec.template and workloadRef satisfy this requirement if synced via GitOps, but workloadRef is recommended because it isolates the pod spec into a single Deployment resource that is easier to keep aligned across clusters. With inline templates, drift between the two Rollout manifests from manual edits or admission webhook mutations silently breaks hash alignment.
ENABLE_PEERING_LABEL_PROPAGATION must be configured on istiod. By default, Solo Enterprise for Istio’s multicluster peering feature does not propagate pod labels to autogenerated WorkloadEntries. Set the pilot environment variable to one of the following:
- all propagates all labels.
- A comma-delimited list of label keys, at minimum app,rollouts-pod-template-hash.
  Enabling label propagation can significantly increase data transferred over the wire between clusters. To minimize overhead, specify only the label keys required for subset matching rather than using all.

Because subset matching depends on hash alignment between clusters, the timing of rollouts on each cluster matters.

During a rollout on cluster1 (cluster2 not yet rolling): cluster2’s pods (still at the old revision’s hash) match cluster1’s stable subset only. Cross-cluster traffic stays on the stable version, which is often the desired behavior while validating the canary locally.
After cluster1’s rollout completes (cluster2 not yet rolled): cluster2’s pods drop out of both subsets until cluster2 also rolls forward. During this window, cross-cluster traffic to the global hostname has no matching endpoints at cluster1’s waypoint.

To minimize this disruption window, kick off rollouts on both clusters in near-sync, ideally driven by the same GitOps reconciliation cycle.

East-west gateways Egress