Skip to content
You are viewing the documentation for Solo Enterprise for Istio, formerly known as Gloo Mesh (OSS APIs).

Add VMs to the mesh

Alpha
Page as Markdown
Onboard workloads that run in a virtual machine to your ambient mesh.
About

As you build your ambient mesh, you might want to add a workload that runs on an external machine to your cluster environment. For example, you might run an app or service in a virtual machine (VM) that must communicate with services in the Istio ambient mesh that runs in your Kubernetes cluster.
To extend the mesh to include workloads running on VMs, you generate a bootstrap token and deploy a ztunnel instance on the VM that uses that token to onboard to your mesh. For VMs with multiple workloads, use the istioctl vm add-workload command to automatically create the necessary Kubernetes resources and per-workload identities. For single-workload VMs, you can use istioctl bootstrap to generate the token manually. Once onboarded, workloads on your VM can communicate bidirectionally with in-mesh services in your cluster via the ztunnel.
VM integration into an ambient mesh is an alpha feature. Alpha features are likely to change, are not fully tested, and are not supported for production. For more information, see Solo feature maturity.
Before you begin

  1. Set up an ambient mesh in a single or multicluster setup.
    If you have not yet set up an ambient mesh, be sure to include the REQUIRE_3P_TOKEN=“false” environment variable in istiod when you follow either of these guides to install an ambient mesh. For details, see the first step in the next section.
  2. Deploy the bookinfo sample app.
  3. If you have not already, get the Solo distribution of Istio binary and install istioctl, which you use for the bootstrap command in this guide.
    1. Save the Solo distribution of Istio version that you installed.
      • Istio 1.29 and later:
        export ISTIO_VERSION=1.30.1
export ISTIO_IMAGE=${ISTIO_VERSION}-solo
export REPO=us-docker.pkg.dev/soloio-img/istio
export HELM_REPO=us-docker.pkg.dev/soloio-img/istio-helm
      • Istio 1.28 and earlier:
Save the repo key for the minor version of the Solo distribution of Istio. This value is the 12-character hash at the end of the repo URL us-docker.pkg.dev/gloo-mesh/istio-<repo-key>, which you can find in the Istio images built by Solo.io support article.
        export ISTIO_VERSION=1.30.1
export ISTIO_IMAGE=${ISTIO_VERSION}-solo
# 12-character hash at the end of the repo URL
export REPO_KEY=<repo_key>
export REPO=us-docker.pkg.dev/gloo-mesh/istio-${REPO_KEY}
export HELM_REPO=us-docker.pkg.dev/gloo-mesh/istio-helm-${REPO_KEY}
    2. Download the Solo distribution of Istio binary and install istioctl. This script automatically detects your OS and architecture, downloads the appropriate Solo distribution of Istio binary, and verifies the installation.
      bash <(curl -sSfL https://raw.githubusercontent.com/solo-io/doc-examples/main/istio/install-istioctl.sh)
export PATH=${HOME}/.istioctl/bin:${PATH}
  4. If you haven’t already, create an east-west gateway to facilitate traffic between the VM, istiod, and other workloads within the mesh.
    1. Create an east-west gateway.
      • Solo distribution of istioctl: For more information about this command, see the CLI reference.
        kubectl create namespace istio-eastwest
istioctl multicluster expose --namespace istio-eastwest --generate > ew-gateway.yaml
kubectl apply -f ew-gateway.yaml
      • Helm: For more information about the peering chart, see the Helm values reference. For recommendations on customizing the east-west gateway for resiliency and availability with the Helm chart, see the best practices for multicluster peering.
        helm upgrade -i peering-eastwest oci://${HELM_REPO}/peering \
  --version ${ISTIO_IMAGE} \
  --namespace istio-eastwest \
  --create-namespace \
  -f - <<EOF
eastwest:
  create: true
  cluster: ${CLUSTER_NAME}
     # The network that the istio-system namespace is labeled with.
     # In prod environments, network and cluster are likely not the same value.
  network: ${CLUSTER_NAME}
  deployment: {}
EOF
    2. Verify that the east-west gateway is successfully deployed.
      kubectl get pods -n istio-eastwest
      Example output:
      NAME                              READY   STATUS    RESTARTS   AGE
istio-eastwest-5d4f757664-6hw7b   1/1     Running   0          9s
  5. Install docker on the VM, which you will use to run to run ztunnel as a container alongside the application.
  6. Choose the steps that match your deployment scenario:
    • Multiple workloads (recommended): The VM runs one or more applications that each get their own mesh identity. Note that this approach uses istioctl vm add-workload and is the recommended configuration even for single-workload VMs, because it is more flexible as deployment requirements change.
    • Single workload (legacy): A limited, manual approach using istioctl bootstrap for VMs that run a single application.
Multi-workload VM (recommended)

This is the recommended approach for onboarding VMs to the ambient mesh, even if the VM currently runs only a single application. Each workload on the VM gets its own SPIFFE identity, certificates, and authorization policy enforcement, so that you can add workloads later without re-onboarding the VM. This method also gives you the flexibility to target policies to specific workload identities.
Ztunnel runs on the VM in a shared proxy mode. A gateway WorkloadEntry represents the VM itself, and each application gets its own WorkloadEntry with a per-workload identity. Only a single HBONE port (15008) is required on the host.
This feature is available in Solo Enterprise for Istio version 1.30 and later.
Before you begin: Complete the Before you begin prerequisites.
Onboard the VM and its workloads

  1. Save the VM namespace, IP address, and hostname as environment variables. The IP address can be a private or public IP, but it must be reachable by other mesh workloads.
    export VM_NAMESPACE=vm-apps
export VM_IP=<VM_IP_address>
export VM_HOSTNAME=$(ssh <vm-user>@${VM_IP} hostname)
  2. Create the namespace and label it for ambient mesh inclusion.
    kubectl create namespace ${VM_NAMESPACE}
kubectl label namespace ${VM_NAMESPACE} istio.io/dataplane-mode=ambient
  3. Add the workloads and start ztunnel using one of the following approaches.
    Run istioctl vm add-workload for each workload on the VM. The first invocation creates the gateway WorkloadEntry and ServiceAccount and outputs the bootstrap token needed to start ztunnel. Subsequent invocations add resources for additional workloads without regenerating the token.
    1. Add the first workload. Use --output-dir to write the per-workload token to a file. For more information about this command, run istioctl vm add-workload --help or see the CLI reference.
      istioctl vm add-workload app1 \
  --external \
  --address ${VM_IP} \
  --namespace ${VM_NAMESPACE} \
  --ports http:80:8080 \
  --hostname ${VM_HOSTNAME} \
  --output-dir ./vm-tokens
      Example output:
      Created namespace "vm-apps"
Configured service account vm-apps/app1
Created WorkloadEntry vm-apps/vm-app1
Configured gateway service account vm-apps/vm-gateway
Created gateway WorkloadEntry vm-apps/vm-<hostname>-gateway

Start ztunnel on the VM with:
  BOOTSTRAP_TOKEN=<generated_token> ztunnel
    2. Save the BOOTSTRAP_TOKEN value from the output, which is needed in later steps to start ztunnel on the VM.
      export BOOTSTRAP_TOKEN=<generated_token>
    3. Add the second workload. The gateway already exists from the previous command, so no new bootstrap token is generated.
      istioctl vm add-workload app2 \
  --address ${VM_IP} \
  --namespace ${VM_NAMESPACE} \
  --ports http:80:9090 \
  --output-dir ./vm-tokens
    4. Distribute the workload tokens to the VM. Each token must exist at the path /etc/ztunnel/tokens/<namespace>/<workload-name>/token.
      ssh <vm-user>@${VM_IP} "sudo mkdir -p /etc/ztunnel/tokens/${VM_NAMESPACE}/app1 /etc/ztunnel/tokens/${VM_NAMESPACE}/app2"
scp ./vm-tokens/app1.token <vm-user>@${VM_IP}:/etc/ztunnel/tokens/${VM_NAMESPACE}/app1/token
scp ./vm-tokens/app2.token <vm-user>@${VM_IP}:/etc/ztunnel/tokens/${VM_NAMESPACE}/app2/token
    5. SSH into the VM and start ztunnel by using the bootstrap token from the output of step 3a. This command mounts the per-workload token directory so ztunnel can discover each workload’s identity.
      docker run -d \
  --name ztunnel \
  --network host \
  -e BOOTSTRAP_TOKEN="${BOOTSTRAP_TOKEN}" \
  -v /etc/ztunnel:/etc/ztunnel:ro \
  us-docker.pkg.dev/soloio-img/istio/ztunnel:1.30.1-solo
     

    Alternatively, if you installed the ztunnel binary directly on the VM, you can start it with BOOTSTRAP_TOKEN=${BOOTSTRAP_TOKEN} ztunnel.
    1. Save a unique identifier for the VM. This value must be unique within the network and must match the NODE_NAME you set when you start ztunnel.
      export VM_NODE_NAME=<unique-vm-identifier>
    2. Create a ServiceAccount for the gateway identity and a gateway WorkloadEntry.
      kubectl create serviceaccount vm-gateway -n ${VM_NAMESPACE}
      kubectl apply -n ${VM_NAMESPACE} -f - << EOF
apiVersion: networking.istio.io/v1
kind: WorkloadEntry
metadata:
  name: vm-gateway
  namespace: ${VM_NAMESPACE}
spec:
  address: ${VM_IP}
  serviceAccount: vm-gateway
EOF
    3. Create a WorkloadEntry, Service, and ServiceAccount for each workload. Each WorkloadEntry uses two annotations that tell istiod how to route traffic to it:
      • solo.io/network-gateway: References the gateway WorkloadEntry. Istiod routes all ztunnel traffic through the VM gateway instead of connecting directly.
      • solo.io/node: A unique identifier for the VM, matching the NODE_NAME set when starting ztunnel. Istiod uses this to provision per-workload certificate managers and inbound listeners.
      kubectl apply -n ${VM_NAMESPACE} -f - << EOF
apiVersion: networking.istio.io/v1
kind: WorkloadEntry
metadata:
  name: vm-app1
  namespace: ${VM_NAMESPACE}
  annotations:
    solo.io/network-gateway: vm-gateway
    solo.io/node: ${VM_NODE_NAME}
  labels:
    app: app1
spec:
  address: 127.0.0.1
  ports:
    http: 8080
  serviceAccount: vm-app1
---
apiVersion: v1
kind: Service
metadata:
  name: app1
  namespace: ${VM_NAMESPACE}
spec:
  ports:
  - name: http
    port: 80
    targetPort: 8080
  selector:
    app: app1
---
apiVersion: networking.istio.io/v1
kind: WorkloadEntry
metadata:
  name: vm-app2
  namespace: ${VM_NAMESPACE}
  annotations:
    solo.io/network-gateway: vm-gateway
    solo.io/node: ${VM_NODE_NAME}
  labels:
    app: app2
spec:
  address: 127.0.0.1
  ports:
    http: 9090
  serviceAccount: vm-app2
---
apiVersion: v1
kind: Service
metadata:
  name: app2
  namespace: ${VM_NAMESPACE}
spec:
  ports:
  - name: http
    port: 80
    targetPort: 9090
  selector:
    app: app2
EOF
      kubectl create serviceaccount vm-app1 -n ${VM_NAMESPACE}
kubectl create serviceaccount vm-app2 -n ${VM_NAMESPACE}
    4. Generate a ServiceAccount token for each workload and copy it to the VM at the path /etc/ztunnel/tokens/<namespace>/<workload-name>/token. Note that kubectl create token generates short-lived tokens. For production environments, use a longer token expiry or a token rotation process.
      kubectl create token vm-app1 -n ${VM_NAMESPACE} > /tmp/vm-app1-token
ssh <vm-user>@${VM_IP} "sudo mkdir -p /etc/ztunnel/tokens/${VM_NAMESPACE}/vm-app1"
scp /tmp/vm-app1-token <vm-user>@${VM_IP}:/etc/ztunnel/tokens/${VM_NAMESPACE}/vm-app1/token

kubectl create token vm-app2 -n ${VM_NAMESPACE} > /tmp/vm-app2-token
ssh <vm-user>@${VM_IP} "sudo mkdir -p /etc/ztunnel/tokens/${VM_NAMESPACE}/vm-app2"
scp /tmp/vm-app2-token <vm-user>@${VM_IP}:/etc/ztunnel/tokens/${VM_NAMESPACE}/vm-app2/token
    5. Generate a bootstrap token for the VM gateway identity.
      istioctl bootstrap --namespace ${VM_NAMESPACE} --service-account vm-gateway
    6. SSH into the VM and start ztunnel with PROXY_MODE=shared_vm and the NODE_NAME that matches the solo.io/node annotation on your WorkloadEntries. This command mounts the per-workload token directory so ztunnel can discover each workload’s identity.
      docker run -d \
  --name ztunnel \
  --network host \
  -e BOOTSTRAP_TOKEN="${BOOTSTRAP_TOKEN}" \
  -e PROXY_MODE=shared_vm \
  -e NODE_NAME=${VM_NODE_NAME} \
  -v /etc/ztunnel:/etc/ztunnel:ro \
  us-docker.pkg.dev/soloio-img/istio/ztunnel:1.30.1-solo
     

    Alternatively, if you installed the ztunnel binary directly on the VM, you can start it with BOOTSTRAP_TOKEN=${BOOTSTRAP_TOKEN} PROXY_MODE=shared_vm NODE_NAME=${VM_NODE_NAME} ztunnel.
    Ztunnel automatically discovers its local workloads from XDS. When istiod pushes workload resources matching solo.io/node: ${VM_NODE_NAME}, ztunnel dynamically provisions a certificate manager, an inbound listener, and a SOCKS5 username mapping for each workload. No restart is required when workloads are added or removed.
Route traffic to the VM workloads

The WorkloadEntries and Services created in Step 1 enable other mesh workloads to reach the VM by using a standard Kubernetes service hostname. Verify that all resources exist before testing connectivity.
  1. Verify that the WorkloadEntries and Services were created in the VM namespace.
    kubectl get workloadentries -n ${VM_NAMESPACE}
kubectl get svc -n ${VM_NAMESPACE}
    Example output:
    NAME                     AGE
vm-app1                  30s
vm-app2                  30s
vm-<hostname>-gateway    30s

NAME    TYPE        CLUSTER-IP      EXTERNAL-IP   PORT(S)   AGE
app1    ClusterIP   10.96.1.10      <none>        80/TCP    30s
app2    ClusterIP   10.96.1.11      <none>        80/TCP    30s
  2. Note the in-mesh hostnames for each workload. After the resources are created, pods in the ambient mesh can reach each VM workload at <workload-name>.${VM_NAMESPACE}.svc.cluster.local. For example:
    • app1.${VM_NAMESPACE}.svc.cluster.local
    • app2.${VM_NAMESPACE}.svc.cluster.local
  3. To test outbound connectivity from the VM to in-mesh services, SSH into the VM and use the SOCKS5 proxy to send requests as each workload’s identity. A 200 OK response indicates that the VM workload has successfully joined the mesh and can communicate with in-mesh services.
    ALL_PROXY=socks5h://app1.${VM_NAMESPACE}:pass@127.0.0.1:15080 curl productpage.bookinfo:9080
ALL_PROXY=socks5h://app2.${VM_NAMESPACE}:pass@127.0.0.1:15080 curl productpage.bookinfo:9080
  4. Test inbound connectivity from Kubernetes pods to the VM workloads.
    1. From your cluster, open an interactive shell in a temporary client pod.
      kubectl run client -n ${VM_NAMESPACE} --image=curlimages/curl --rm -it --restart=Never -- sh
    2. From inside the client pod, send requests to each VM workload using its in-mesh hostname.
      curl app1.${VM_NAMESPACE}
curl app2.${VM_NAMESPACE}
      Successful responses from both workloads confirm that inbound traffic from the mesh reaches each workload independently through the VM gateway.
Single-workload VM (legacy)

This method is a limited, manual approach for VMs that run a single application. It uses istioctl bootstrap and requires you to create the Kubernetes routing resources yourself. For a more flexible approach that supports adding workloads later, see Multi-workload VM.
Before you begin: Complete the Before you begin prerequisites.
Onboard the VM and its workload

  1. If you haven’t already, update your istiod installation to add the REQUIRE_3P_TOKEN="false" environment variable on istiod, which is required for the ztunnel that you deploy to the VM in later steps to connect to istiod. In a multicluster mesh setup, enable this environment variable on the istiod installation in the cluster you want to connect the VM to.
    Create the following gloo-extensions-config configmap.
    kubectl apply -n gloo-mesh -f -<<EOF
apiVersion: v1
kind: ConfigMap
metadata:
  name: gloo-extensions-config
  namespace: gloo-mesh
data:
  values.istiod: |
    env:
      REQUIRE_3P_TOKEN: false
EOF
    Get the current values for the istiod Helm release and upgrade it with the --set env.REQUIRE_3P_TOKEN="false" setting.
    helm get values istiod -n istio-system -o yaml > istiod.yaml
helm upgrade istiod oci://${HELM_REPO}/istiod \
-n istio-system \
-f istiod.yaml \
--set env.REQUIRE_3P_TOKEN="false"
  2. Save the VM name, namespace, service account, and IP address. The IP address can be a private or public IP, but it must be reachable by other mesh workloads. The name, namespace, and service account can be named whatever you prefer. The service account represents the VM in the cluster so that Istio control plane manages it the same way as any other pod in the mesh. If you later want to apply Istio resources to your VM workload, you can use this service account and namespace in the configuration.
    export VM_NAME=vm-example
export VM_NAMESPACE=vm-ns
export VM_SERVICE_ACCOUNT=vm-sa
export VM_IP=<VM_IP_address>
  3. In your cluster, generate an Istio bootstrap configuration.
    • This command creates a bootstrap token that includes the necessary certificates and metadata for the VM to join the ambient mesh. ztunnel will use this token to authenticate with istio.
    • For more information about this command, run istioctl bootstrap --help or see the CLI reference.
    kubectl create namespace ${VM_NAMESPACE}
kubectl label namespace ${VM_NAMESPACE} istio.io/dataplane-mode=ambient
kubectl --namespace ${VM_NAMESPACE} create serviceaccount ${VM_SERVICE_ACCOUNT}
istioctl bootstrap --namespace ${VM_NAMESPACE} --service-account ${VM_SERVICE_ACCOUNT}
  4. SSH into the VM to save the bootstrap token that you generated as an environment variable.
    export BOOTSTRAP_TOKEN=<generated_token>
  5. Start a ztunnel instance on the VM. Ztunnel is a lightweight data plane component that enables the VM to participate in the ambient mesh. This command pulls the ztunnel container image and starts it with the necessary configuration to connect to the mesh.
    docker run -d -e BOOTSTRAP_TOKEN=${BOOTSTRAP_TOKEN} --network=host us-docker.pkg.dev/soloio-img/istio/ztunnel:1.30.1-solo
  6. Test connectivity from the VM to services in the mesh, such as to the productpage service in the bookinfo namespace. For example, the following curl commands test connectivity by using productpage’s Kubernetes DNS name and mesh-internal DNS name. Two 200 OK responses indicate that the VM has successfully joined the mesh and can communicate with other in-mesh services.
    export ALL_PROXY=socks5h://127.0.0.1:15080
curl productpage.bookinfo:9080
curl productpage.bookinfo.mesh.internal:9080
Route traffic to the VM

To route traffic from other mesh workloads to the VM, create a ServiceEntry and WorkloadEntry to represent the VM within the mesh.
  1. Save the following routing details in environment variables.
    # The port that clients in the mesh route to
export CLIENT_PORT=80
# The port ztunnel proxies inbound requests to on the VM
export APPLICATION_PORT=8080
# The Istio network name
export NETWORK=$(kubectl get namespace istio-system -o jsonpath='{.metadata.labels.topology\.istio\.io\/network}')
  2. Create a ServiceEntry and WorkloadEntry to represent the VM within the mesh. After you apply these resources, the ambient mesh now routes traffic from Kubernetes pods to the VM via the address vm-example.vm-ns.svc.cluster.local.
    kubectl apply -n ${VM_NAMESPACE} -f - << EOF
apiVersion: networking.istio.io/v1beta1
kind: ServiceEntry
metadata:
  name: ${VM_NAME}
  namespace: ${VM_NAMESPACE}
spec:
  hosts:
  - ${VM_NAME}.${VM_NAMESPACE}.svc.cluster.local
  ports:
  - number: ${CLIENT_PORT}
    name: http
    protocol: HTTP
    targetPort: ${APPLICATION_PORT} 
  resolution: STATIC 
  workloadSelector: 
    labels:
      app: ${VM_NAME}
---
apiVersion: networking.istio.io/v1beta1
kind: WorkloadEntry
metadata:
  name: ${VM_NAME}
  namespace: ${VM_NAMESPACE}
  labels:
    app: ${VM_NAME}
  annotations:
    ambient.istio.io/redirection: enabled
spec:
  address: ${VM_IP}
  serviceAccount: ${VM_SERVICE_ACCOUNT}
  network: ${NETWORK}
EOF
Traffic flows and limitations

Outbound traffic from the VM

Applications on the VM use the SOCKS5 proxy at port 15080 for outbound requests. To send traffic as a specific workload identity, set the SOCKS5 username to <workload-name>.<namespace>, where the workload name matches the ServiceAccount name.
# Send a request as app1's identity
ALL_PROXY=socks5h://app1.${VM_NAMESPACE}:pass@127.0.0.1:15080 curl http://<service>.<namespace>

# Send a request as app2's identity
ALL_PROXY=socks5h://app2.${VM_NAMESPACE}:pass@127.0.0.1:15080 curl http://<service>.<namespace>
If no username is provided, ztunnel uses the VM gateway identity for outbound connections.
Inbound traffic to the VM

Kubernetes pods reaching a VM workload go through a double-HBONE flow:
  1. The client ztunnel sees the solo.io/network-gateway annotation and establishes double-HBONE to the VM gateway at port 15008.
  2. The VM gateway ztunnel terminates the outer HBONE layer.
  3. The gateway routes the inner HBONE to the workload’s dedicated inbound listener (assigned sequentially from port 15100).
  4. The workload-specific inbound listener terminates the inner HBONE with the workload’s identity.
  5. Plain TCP reaches the application on localhost.
Authorization policies target the workload’s ServiceAccount identity (spiffe://cluster.local/ns/${VM_NAMESPACE}/sa/<workload-name>) and work the same as for any in-cluster pod.
Per-workload authorization

Because each workload on a multi-workload VM has its own ServiceAccount, you can write authorization policies that target a specific workload’s SPIFFE identity without affecting other workloads on the same instance.
By identity (principal): Target a workload by its SPIFFE identity. This is useful for controlling which workloads can reach a destination, regardless of where they run.
kubectl apply -f - <<EOF
apiVersion: security.istio.io/v1
kind: AuthorizationPolicy
metadata:
  name: httpbin-deny-app2
  namespace: ${VM_NAMESPACE}
spec:
  selector:
    matchLabels:
      run: httpbin
  action: DENY
  rules:
  - from:
    - source:
        principals: ["cluster.local/ns/${VM_NAMESPACE}/sa/app2"]
EOF
This policy denies app2 from reaching httpbin while leaving app1 (and any other identity) unaffected.
By workload label: The istioctl vm add-workload command applies the vm.solo.io/workload label to each WorkloadEntry it creates, using the workload name as the value. You can use this label to target inbound traffic to a specific workload on the VM.
kubectl apply -f - <<EOF
apiVersion: security.istio.io/v1
kind: AuthorizationPolicy
metadata:
  name: deny-inbound-app1
  namespace: ${VM_NAMESPACE}
spec:
  selector:
    matchLabels:
      vm.solo.io/workload: app1
  action: DENY
  rules:
  - from:
    - source:
        namespaces: ["${VM_NAMESPACE}"]
EOF
This policy denies inbound traffic to app1 while leaving other workloads on the same VM unaffected.
Limitations

  • Applications must support SOCKS5 for outbound traffic (ALL_PROXY environment variable).
  • Per-workload ServiceAccount tokens must be distributed to the VM manually and rotated before expiry.
  • Transparent traffic interception is not supported.
  • Multi-workload VM support does not apply to EC2-backed ECS clusters. This support is planned for a future release.
Cleanup

  1. Stop and remove ztunnel on the VM.
    docker stop ztunnel && docker rm ztunnel
  2. Remove the VM namespace and all resources within it from your cluster.
    kubectl delete namespace ${VM_NAMESPACE}
  3. If you created an east-west gateway for this guide, you can optionally remove it.
    kubectl delete namespace istio-eastwest
Add EC2 instances to the mesh