Migrate from a sidecar mesh ALPHA

Considerations link

The Solo ambient migration tool provides a prescriptive path to help you migrate from Istio sidecar mode to ambient mode. Before you begin the migration process, review the following considerations:

Ambient architecture: The tool walks you through each step of migrating your workloads to an ambient mesh. However, familiarizing yourself with ambient mode and its differences from sidecar mode is recommended.

• Review the conceptual documentation to get an overview of the ambient architecture and key conceptual differences.
• If you use authorization policies, understanding how authorization policies work in an ambient mesh is crucial before you begin the migration.

Platform prerequisites for ambient: Be sure to check the platform-specific prerequisites for ambient to determine whether you must make any changes to your environment before you migrate an ambient mesh.

Using the migration tool link

1. Install the gloo CLI client, which contains the migration tool.

     curl -sL https://storage.googleapis.com/gloo-cli/install.sh | sh -
   
   export PATH=$HOME/.gloo/bin:$PATH
     

2. Optional: Before you start the migration, you can use the free Ambient Estimator Tool to assess your Istio environment and estimate potential cost savings.

   1. Gather your cluster resource usage data and save it in a cluster configuration file. You can optionally change the output directory with the --output-dir flag, and hide the names of the cluster and namespaces by using a hash with the --hide-names flag.

        gloo ambient estimate
        

   2. Upload the output file to the Ambient Estimator Tool to break down potential cost savings for your cluster setup.

3. Start the migration process by running the gloo ambient migrate command. This command analyzes the state of the cluster that your kubeconfig context currently points to, and recommends next steps in multiple phases.

     gloo ambient migrate
     

   Review the following optional flags you can specify in the migrate command.

   Option	Description
   --enterprise	Include this flag if you have a Gloo Mesh Enterprise license, which is required for some ambient features that are available in the Solo distribution of Istio. If this flag is included, checks for those features in your cluster pass.
   --from-files	Read configuration from YAML files instead of a live cluster, such as to test the migration first. When providing files, be sure to include the following APIs: All Istio APIs All Gateway APIs Kubernetes APIs: Pod DaemonSet Deployment ValidatingWebhookConfiguration Namespace Service ConfigMap Secret (optional)
   --ignore-failures	Run the tool through each phase, regardless of whether checks fail in any phases. This can be helpful during testing if you are aware of failures you want to ignore.
   --output-dir	Directory to output generated files to, such as the recommended changes to policies and waypoint configuration.

Migration phases link

Each phase of the migration runs a series of checks, and validates that the checks are completed before proceeding. If checks in a phase fail, the tool gives recommendations for how to resolve them.

Phase 1: Prerequisites link

The pre-reqs phase inspects the cluster to ensure that it is ready for ambient mode.

This includes ensuring that the Kubernetes environment, the current Istio version, and features used by the existing Istio installation are compatible. For example, all existing Istio installations in the cluster:

• Must run Istio version 1.25 or later
• For zero-downtime migrations, must run the Solo distribution of Istio

Note that if an Istio version upgrade to 1.25 or later is required, you can optionally follow these steps to also enable ambient mode for your Istio installation during the upgrade. Otherwise, you can upgrade your version in this phase, and enable ambient mode in the next phase.

Example output for a cluster that is ready for ambient mode:

  • Starting phase pre-reqs...
✅ Phase pre-reqs succeeded!
  ✅ Cluster CNI compatibility: passed
  ✅ Istio version compatibility: passed
  ✅ Multicluster usage compatibility: passed
  ✅ Virtual Machine usage compatibility: passed
  ✅ SPIRE usage compatibility: passed
  

Phase 2: Cluster setup link

The cluster-setup phase ensures that your Istio installation is installed in ambient mode.

Switching your installation to ambient mode allows you to later enroll workloads in the ambient mesh, and enables existing sidecar and gateways in the dataplane to communicate with ambient workloads.

Example output for a cluster that has the Kubernetes Gateway API CRDs installed, but does not yet have ambient mode enabled for istiod, and does not have the CNI and ztunnel components:

  • Starting phase cluster-setup...
❌ Phase cluster-setup failed!
  ❌ Ambient mode enabled: failed. 1 error occurred:
    * istiod istio-system/istiod-main must have 'PILOT_ENABLE_AMBIENT=true'. Upgrade Istio with '--set profile=ambient'.
  
  
  ❌ DaemonSets deployed: failed. 2 errors occurred:
    * ztunnel not found
    * Istio CNI agent not found
  
  
  ✅ Sidecars support ambient mode: passed
  ✅ Required CRDs installed: passed
  

Enabling ambient mode link

To upgrade your sidecar Istio installation to ambient mode, you can follow these steps. For more information, such as expanded examples of recommended Istio installation settings, see the ambient installation guide.

1. Optional: Set your Solo license key as an environment variable. A license unlocks several advanced licensed features, such as multicluster ambient mesh functionality, in addition to the features provided by default in the Solo distribution of Istio. To obtain a license, contact an account representative. If you prefer to specify license keys in a secret instead, see Licensing.

     export LICENSE_KEY=<license_key>
     

2. Save the details for version 1.25 or later of the Solo distribution of Istio.

   1. Save the Solo distribution of Istio patch version and tag.

        export ISTIO_VERSION=1.26.2
      # Change the tags as needed
      export ISTIO_IMAGE=${ISTIO_VERSION}-solo
        

   2. Save the repo key for the minor version of the Solo distribution of Istio that you want to install. This is the 12-character hash at the end of the repo URL us-docker.pkg.dev/gloo-mesh/istio-<repo-key>, which you can find in the Istio images built by Solo.io support article.

        # 12-character hash at the end of the minor version repo URL
      export REPO_KEY=<repo_key>
      export REPO=us-docker.pkg.dev/gloo-mesh/istio-${REPO_KEY}
      export HELM_REPO=us-docker.pkg.dev/gloo-mesh/istio-helm-${REPO_KEY}
        

3. Apply the CRDs for the Kubernetes Gateway API.

     kubectl apply -f https://github.com/kubernetes-sigs/gateway-api/releases/download/v1.2.1/standard-install.yaml
     

4. If you have not already, install or upgrade the base chart.

     helm upgrade -i istio-base oci://${HELM_REPO}/base -n istio-system --version ${ISTIO_IMAGE} --wait
     

5. Get the current values for the istiod Helm release in your cluster.

     helm get values istiod -n istio-system -o yaml > istiod.yaml
     

6. Upgrade your istiod Helm release with the --profile=ambient option, which enables settings required for ambient mode in istiod.

     helm upgrade istiod oci://${HELM_REPO}/istiod \
   -n istio-system \
   --version ${ISTIO_IMAGE} --wait \
   -f istiod.yaml \
   --set profile=ambient \
   --set global.tag=${ISTIO_IMAGE} \
   --set global.hub=${REPO} \
   --set license.value=${LICENSE_KEY}
     

   notifications

   In the Solo distribution of Istio 1.25 and later, you can access enterprise-level features by passing your Solo license in the license.value or license.secretRef field of the Solo distribution of the istiod Helm chart. The Solo istiod Helm chart is strongly recommended due to the included safeguards, default settings, and upgrade handling to ensure a reliable and secure Istio deployment. Though it is not recommended, you can pass your license key in the open source istiod Helm chart by using the --set pilot.env.SOLO_LICENSE_KEY field.
7. Install or upgrade the CNI chart with the --profile=ambient option.

     helm upgrade -i istio-cni oci://${HELM_REPO}/cni \
   -n istio-system \
   --version ${ISTIO_IMAGE} --wait \
   -f - <<EOF
   ambient:
     dnsCapture: true
   excludeNamespaces:
     - istio-system
     - kube-system
   global:
     hub: ${REPO}
     tag: ${ISTIO_IMAGE}
   profile: ambient
   EOF
     

8. Install the ztunnel chart.

     helm install ztunnel oci://${HELM_REPO}/ztunnel -n istio-system \
   --version ${ISTIO_IMAGE} --wait \
   -f - <<EOF
   configValidation: true
   enabled: true
   env:
     L7_ENABLED: "true"
   hub: ${REPO}
   istioNamespace: istio-system
   namespace: istio-system
   profile: ambient
   proxy:
     clusterDomain: cluster.local
   tag: ${ISTIO_IMAGE}
   terminationGracePeriodSeconds: 29
   variant: distroless
   # If you installed istiod with a revision, specify that revision
   #revision: main
   # If you specified a global.multicluster.clusterName in istiod, 
   # specify that same value
   #multiCluster:
   #  clusterName: ${CLUSTER_NAME}
   EOF
     

9. Check the status of your Istio component pods.

     kubectl get pods -n istio-system
     

10. Repeat the migration command to check your setup.

      gloo ambient migrate
      

Phase 3: Waypoint deployment link

The deploy-waypoints phase determines which services require waypoint proxies to be deployed to retain their existing functionality.

Unlike with sidecar mode, ambient mode allows you to choose whether you want to apply policies on Layer 7. In particular, if a service requires rich HTTP based functionality, you can deploy a waypoint proxy. Based on the existing policies in the cluster, the tool provides a best-effort analysis for whether a waypoint is recommended for specific namespaces or services, and the suggested waypoint configuration to deploy.

For best results, be sure to carefully review the tool’s recommendations before deploying waypoints. If you aren’t sure whether to deploy a waypoint, deploying the waypoint is recommended.

Review the following example waypoint scenarios:

• The tool might detect customized load balancing for a service and recommend a waypoint, because waypoints are required to implement custom load balancing policies. However, you might decide that the value of this policy doesn’t justify the waypoint, and choose to skip the recommendation.
• In some cases, you might deploy a waypoint to a namespace even when the tool does not recommend one. For example, even without policies, sidecars provide a variety of functionalities such as mTLS, HTTP observability, and HTTP request-level load balancing. While ztunnel provides mTLS and HTTP observability, it does not provide request-level load balancing. Applications that require this can benefit from a waypoint, even when the tool does not recommend one.

Example output for a cluster in which one waypoint is recommended:

  • Starting phase deploy-waypoints...
⚠️ Phase deploy-waypoints has recommendations!
  🔮 Namespace "application" might require a waypoint for the following services:
     * Service "application/hello-world" depends on VirtualService "application/hello-world"

  ℹ️ Generated waypoints written to /tmp/istio-migrate/recommended-waypoints.yaml
  

The recommended-waypoints.yaml file contains the following recommended waypoint configuration:

  apiVersion: gateway.networking.k8s.io/v1
kind: Gateway
metadata:
  name: waypoint
  namespace: application
spec:
  gatewayClassName: istio-waypoint
  listeners:
  - name: mesh
    port: 15008
    protocol: HBONE
  

For more information about configuring waypoints, see the waypoint proxy guide.

Phase 4: Policy migration link

The migrate-policies phase determines whether policies must be changed before being applied to the waypoint proxies, and gives recommendations for updated policies.

For policies that modify services, VirtualServices and DestinationRules, no changes are required. The tool copies the configuration and generates policies to apply to waypoints. However, policies that modify specific workloads must be modified before they can be applied to a waypoint. The tool provides recommendations on how to modify and apply the following resources to the waypoint:

• Telemetry
• WasmPlugin
• RequestAuthentication
• AuthorizationPolicy
• EnvoyFilter

The tool also detects and warns against policies that are not supported in ambient mode.

In this example output, the configuration for a few authorization policies is copied to new policies that apply the config to the waypoint:

  • Starting phase migrate-policies...
⚠️ Phase migrate-policies has recommendations!
  🔮 Apply AuthorizationPolicy/app1/echo-from-waypoint: Service/app1/echo must allow traffic from its waypoint.
  🔮 Apply AuthorizationPolicy/app1/echo-legacy-policies: Existing configuration is copied from policy app1/legacy-policies to be enforced at the waypoint.
  🔮 Apply AuthorizationPolicy/app1/echo-policies: Existing configuration is copied from policy app1/policies to be enforced at the waypoint.
  🔮 Apply AuthorizationPolicy/app1/waypoint-allow-nothing: Existing configuration is copied from namespace policy app1/allow-nothing to be enforced at the waypoint.
  ℹ️ Recommended policies written to /tmp/istio-migrate/recommended-policies.yaml
  

The recommended-policies.yaml file contains the recommended policies that are generated. Note that at this phase of the process, you apply these waypoint-based policies in addition to the existing sidecar-based policies. Later phases help clean up the original policies once the migration is complete.

Authorization policies link

In particular, authorization policies require careful handling due to the changes in how policies are enforced in ambient mode. In ambient mode, Layer 4 and Layer 7 policies can be enforced against services, applied at the waypoint. Additionally, Layer 4 policies can be enforced against workloads.

This requires the following changes:

• If a workload policy contains HTTP attributes, a waypoint is required, as recommended in the previous phase. The existing policy must then be moved to configure the waypoint rather than the workload.
• If a waypoint exists, the workload must allow traffic from the waypoint.

The tool helps copy the authorization policies to the waypoints as required, and additionally generates policies to allow traffic from the waypoints.

Applying recommended policies link

Note that all policy recommendations that the tool provides are intended to be applied in addition to the existing policies. This ensures that during the migration process, policies continue to apply to all traffic requests, regardless of whether traffic is routed through a waypoint. Later phases then transition traffic to start flowing through the waypoint, and optionally provide recommendations for cleaning up the old, unused policies.

For example, consider a case where authorization policies do not allow any requests to apps in ns-bar, except from apps in ns-foo to app-bar:

  apiVersion: security.istio.io/v1
kind: AuthorizationPolicy
metadata:
  name: allow-nothing
  namespace: ns-bar
spec:
  {}
---
apiVersion: security.istio.io/v1
kind: AuthorizationPolicy
metadata:
  name: legacy-policy
  namespace: ns-bar
spec:
  selector:
    matchLabels:
      app: app-bar
  rules:
    - from:
      - source:
          namespaces: ["ns-foo"]
  

When you deploy a waypoint, the tool recommends additional policies to support the same behavior for traffic routed from the waypoint, in addition to traffic that is not routed through the waypoint.

For example, the following authorization policies are generated, which replicate the same behavior. No traffic is permitted to apps in ns-bar, except for requests directly routed by the namespace’s waypoint to app-bar. Then, the waypoint permits requests only from apps in ns-foo to app-bar.

  # Existing configuration from namespace policy ns-bar/allow-nothing is copied to be enforced at the waypoint.
apiVersion: security.istio.io/v1
kind: AuthorizationPolicy
metadata:
  annotations:
    migrate.solo.io/source-policy: ns-bar/allow-nothing
  name: waypoint-allow-nothing
  namespace: ns-bar
spec:
  targetRefs:
  - group: gateway.networking.k8s.io
    kind: Gateway
    name: waypoint
---
# v1/Service/ns-bar/app-bar must allow traffic from its waypoint.
apiVersion: security.istio.io/v1
kind: AuthorizationPolicy
metadata:
  annotations:
    migrate.solo.io/source-service: app-bar
  name: app-bar-from-waypoint
  namespace: ns-bar
spec:
  rules:
  - from:
    - source:
        serviceAccounts:
        - waypoint
  selector:
    matchLabels:
      app: app-bar
---
# Existing configuration from policy ns-bar/legacy-policy is copied to be enforced at the waypoint.
apiVersion: security.istio.io/v1
kind: AuthorizationPolicy
metadata:
  annotations:
    migrate.solo.io/source-policy: ns-bar/legacy-policy
  name: app-bar-policies
  namespace: ns-bar
spec:
  rules:
  - from:
    - source:
        namespaces:
        - ns-foo
  targetRefs:
  - kind: Service
    name: app-bar
  

When you apply the recommended policies, the two sets of policies ensure that restrictions on traffic requests for app-bar are enforced on both sidecar and waypoint paths to maintain a zero-downtime transition.

Phase 5: Waypoint enablement link

The use-waypoints phase detects which waypoints must be enabled and provides steps to enable them.

After a waypoint is enabled, traffic starts to flow through the waypoint. Because sidecar-ambient interoperability is supported in the Solo distribution of Istio, traffic flows from waypoints to the Istio sidecars that your workloads still have. Enablement can be incrementally done on a service-by-service basis, which allows you to monitor the health of the system as you progress. If a service behaves unexpectedly, you can revert the waypoint enablement. For more information, see the waypoint configuration documentation in the ambient mesh docs.

Example output, in which the waypoint that was previously created for services in namespace app1 is ready to be enabled:

  • Starting phase use-waypoints...
⚠️ Phase use-waypoints has recommendations!
  ⚠️ Waypoint app1/waypoint is not used by any services.
  🔮 Namespace app1 requires a waypoint, but is not configured to use one. To configure it: kubectl label namespace %v istio.io/use-waypoint=waypoint
  

Considerations for waypoint enablement link

For most cases, enabling all waypoints is recommended before you begin removing sidecars from workloads. However, this is not strictly required, which might be useful in scenarios where some workloads might take a while to complete migration.

To ensure that all policies are correctly applied, the request flow needs either:

• A sidecar on the client and server side, or
• A waypoint proxy enabled

For this reason, it is not recommended to simply remove sidecars from a namespace once that namespace has a waypoint enabled, because services in that namespace might call services in other namespaces that do not yet have an enabled waypoint.

The following diagram illustrates how a client sidecar successfully communicates with a migrated namespace and with a namespace that is not yet migrated. If you were to remove the client sidecar before the unmigrated namespace is migrated, some policies would not be correctly applied to traffic requests.

Phase 6: Policy simplification link

The policy-simplification phase suggests the safe removal of policies that are specific to sidecars.

Because traffic now flows through your enabled waypoints, the original sidecar-based policies that you replicated to apply to waypoints in the migrate-policies phase no longer apply to any traffic. The tool provides the steps to safely clean up the old policies that are now unused.

Example output in which three sidecar-based authorization policies are ready for deletion:

  • Starting phase policy-simplification...
⚠️ Phase policy-simplification has recommendations!
  🔮 AuthorizationPolicy is migrated and can be deleted: kubectl delete authorizationpolicies.security.istio.io -n app1 allow-nothing
  🔮 AuthorizationPolicy is migrated and can be deleted: kubectl delete authorizationpolicies.security.istio.io -n app1 legacy-policies
  🔮 AuthorizationPolicy is migrated and can be deleted: kubectl delete authorizationpolicies.security.istio.io -n app1 policies
  

Phase 7: Sidecar removal link

The final migration phase, remove-sidecars, recommends steps to remove sidecars from any workloads that still have them, and to moving those workloads to the ambient mesh.

At this point, traffic is configured to route through waypoints for all workloads that require them, and the waypoints are configured to apply equivalent policy configurations. It is now safe to remove sidecars from workloads, and if you have not already done so, label the workloads to be included in the ambient mesh.

Example output for removing the sidecar from one app:

  • Starting phase remove-sidecars...
⚠️ Phase remove-sidecars has recommendations!
  🔮 Sidecars can be safely disabled for namespace app1: kubectl label ns app1 istio-injection- istio.io/dataplane-mode=ambient
  ⚠️ Sidecar detected for pod app1/echo-7fb78cb7c5-plsjl
  

After you finish this phase, the migration is complete!

Next link

• Add apps to the ambient mesh.
• Link meshes together to create a multicluster mesh.