Onboard VMs

Onboard a virtual machine your Gloo Mesh Enterprise environment by using meshctl.

As you build your Gloo Mesh environment, you might want to add a virtual machine to your setup. For example, you might run an app or service in a VM that you want to add to the Istio service mesh within a cluster in your Gloo Mesh setup. This guide walks you through exposing the Istio service mesh on a cluster to your VM, and using meshctl commands to onboard the VM to your service mesh. For more information, see the high-level architectural overview of Istio’s virtual machine integration.

Before you begin

  1. Install the following CLI tools.

    • istioctl, the Istio command line tool. The resources in the guide use Istio version 1.13.5.
    • meshctl, the Gloo Mesh command line tool for bootstrapping Gloo Mesh, registering clusters, describing configured resources, and more. This guide requires version 2.0 or later. To quickly install meshctl:
      curl -sL http://run.solo.io/meshctl/install | sh
  2. Install Gloo Mesh Enterprise into a management cluster.

  3. Register each workload cluster with Gloo Mesh.

  4. Prepare a virtual machine that meets the following requirements.

    • Currently, Debian and CentOS 8 images are supported. The VM in this guide's examples uses a Debian GNU/Linux 10 (Buster) image and runs in Google Cloud Platform (GCP).
    • Enable the VM for SSH access, copying files, and running Istio. If you do not have SSH permissions set for the VM, after you install Istio, be sure to follow the steps in Onboard without SSH to manually onboard the VM to your Gloo Mesh setup.
    • If you use a VM that runs in GCP, ensure that the “Allow HTTP traffic” option is selected under the Firewall section when you create the VM.

Step 1: Install the Istio control plane and east-west gateway in a workload cluster

Set up the Istio control plane and an east-west gateway in a workload cluster to ensure that it can be accessed by your VM. Note that even if you already installed Istio in a cluster, review these steps to ensure you implement the settings that are required for VM onboarding.

  1. Save the following environment variables for use with Istio installation.

    • For REPO, use a Gloo Mesh Istio repo key that you can get by logging in to the Support Center and reviewing the Istio images built by Solo.io support article. Or, for Istio 1.11 or earlier, you can use gcr.io/istio-enterprise. For more information, see Get the Gloo Mesh Istio version that you want to use.
    • For ISTIO_IMAGE, save the version that you downloaded, such as 1.13.5, and append the solo tag, which is required to use many Gloo Mesh Enterprise features such as Gloo Mesh Gateway. You can optionally append other Gloo Mesh Istio tags, as described in About Gloo Mesh Istio. If you downloaded a different version than the following, make sure to specify that version instead.
    • For REVISION, take the Istio version number and replace the periods with hyphens, such as 1.13.5 to 1-13-5.
    • For CLUSTER_NAME, save the name of the workload cluster that you want to install Istio into.
    export REPO=<repo-key>
    export ISTIO_IMAGE=1.13.5-solo
    export REVISION=1-13-5
  2. Install the Istio control plane in a workload cluster that you registered with Gloo Mesh. For example, follow steps 1 and 2 in Deploy Istio in production.

    When you follow the guides to install Istio in your workload cluster, be sure to include the following settings that are required for onboarding a VM. Or, if you already installed Istio, you must update your IstioOperator installation to include the following settings.

    • Enable autoregistration in your installation by setting PILOT_ENABLE_WORKLOAD_ENTRY_AUTOREGISTRATION to true. Autoregistration allows the istiod control plane to create the workload entries associated with your VM automatically after the VM is onboarded.
    • You can optionally enable automatic health checks for the VM workload entry by setting PILOT_ENABLE_WORKLOAD_ENTRY_HEALTHCHECKS=true to true.
    • Verify that third-party tokens are enabled in your cluster by following the steps describe in the Istio docs. If third-party tokens are not enabled, you must include the --set values.global.jwtPolicy=first-party-jwt setting.
    istioctl install -f <istiod-kubernetes-values>.yaml --set values.pilot.env.PILOT_ENABLE_WORKLOAD_ENTRY_AUTOREGISTRATION=true [--set values.pilot.env.PILOT_ENABLE_WORKLOAD_ENTRY_HEALTHCHECKS=true --set values.global.jwtPolicy=first-party-jwt]
  3. Prepare an IstioOperator resource for an Istio east-west gateway. The LoadBalancer service that exposes the east-west gateway allows the VM to access the cluster's service mesh and be onboarded into the mesh. After the VM is onboarded, all traffic sent to and from the VM goes through the east-west gateway.

    When you create the east-west gateway in your workload cluster, be sure to include the following settings. If you already installed an east-west gateway, such as by following the steps in Deploy Istio east-west gateway you must update your east-west gateway to include the following settings.

    • Ensure that the east-west gateway is deployed to the istio-system namespace. If the gateway is deployed to a different namespace, health checks for the VM workload entry will fail.
    • The gateway must be named istio-eastwestgateway, or health checks for the VM workload entry will fail.
    • The gateway must be externally exposed by a LoadBalancer service.
    • Port 15012 must be exposed on the gateway with the name tls-istiod so that the VM can discover the east-west gateway in this cluster.
    cat <<EOF > ./eastwest-gateway.yaml
    apiVersion: install.istio.io/v1alpha1
    kind: IstioOperator
      name: eastwest-gateway-$REVISION
      namespace: istio-system
      # No control plane components are installed
      profile: empty
      # Solo.io Istio distribution repository; required for Gloo Mesh Istio.
      hub: $REPO
      # The Solo.io Gloo Mesh Istio version
      tag: $ISTIO_IMAGE
      # Istio revision to create resources with
      revision: $REVISION
        # Enable the default east-west gateway
          - name: istio-eastwestgateway
            namespace: istio-system
            enabled: true
              istio: eastwestgateway
              version: $REVISION
              app: istio-eastwestgateway
              # Matches spec.values.global.network in the istiod deployment
              topology.istio.io/network: $CLUSTER_NAME
                maxReplicas: 5
                  - resource:
                      name: cpu
                      targetAverageUtilization: 60
                    type: Resource
                minReplicas: 2
                  apiVersion: apps/v1
                  kind: Deployment
                  name: istio-eastwestgateway
                  maxSurge: 100%
                  maxUnavailable: 25%
                  cpu: 2000m
                  memory: 1024Mi
                  cpu: 100m
                  memory: 40Mi
                # sni-dnat adds the clusters required for AUTO_PASSTHROUGH mode
                # Required by Gloo Mesh for east/west routing
                - name: ISTIO_META_ROUTER_MODE
                  value: "sni-dnat"
                # You will need the service externally exposed to onboard VMs
                type: LoadBalancer
                  # Health check port. For AWS ELBs, this port must be listed first.
                  - name: status-port
                    port: 15021
                    targetPort: 15021
                  # Port for multicluster mTLS passthrough; required for Gloo Mesh east/west routing
                  - port: 15443
                    targetPort: 15443
                    # Gloo Mesh looks for this default name 'tls' on a gateway
                    name: tls
                  # Port for VM onboarding
                  - port: 15012
                    targetPort: 15012
                    # Required for VM onboarding discovery address
                    name: tls-istiod
                - apiVersion: apps/v1
                  kind: Deployment
                  name: istio-eastwestgateway
                    # Sleep 25s on pod shutdown to allow connections to drain
                    - path: spec.template.spec.containers.[name:istio-proxy].lifecycle
                            - sleep
                            - "25"
                    # Schedule pods on separate nodes if possible
                    - path: spec.template.spec.affinity
                          - podAffinityTerm:
                                - key: app
                                  operator: In
                                  - istio-eastwestgateway
                              topologyKey: kubernetes.io/hostname
                            weight: 100
      # Helm values overrides
            # Enable gateway injection
            injectionTemplate: gateway
        # https://istio.io/v1.5/docs/reference/config/installation-options/#global-options
          # Required for connecting VirtualMachines to the mesh
          network: $CLUSTER_NAME
          # Required for annotating Istio metrics with the cluster name.
          # Must match the trustDomain and GLOO_MESH_CLUSTER_NAME
            clusterName: $CLUSTER_NAME
  4. Update the file with the environment values you set earlier, and install the east-west gateway.

    envsubst < eastwest-gateway.yaml > eastwest-gateway-values.yaml
    kubectl apply -f eastwest-gateway-values.yaml
  5. Verify that the east-west gateway load balancer service is assigned an external address.

    kubectl get svc -n istio-system

    Example output:

    NAME                                 TYPE           CLUSTER-IP      EXTERNAL-IP   PORT(S)                                                                                                      AGE
    istio-eastwestgateway            LoadBalancer   <externalip>  status-port:15021β–Ί30075 tls:15443β–Ί30110 tls-istiod:15012β–Ί30907   13s

Step 2: Expose the Istio control plane to the VM

Create a Gateway and VirtualService so that the VM can reach the istiod control plane on your workload cluster through the east-west gateway.

  1. Download the Istio gateway and virtual service resources that allow passthrough to istiod.

    curl -0L https://raw.githubusercontent.com/istio/istio/master/samples/multicluster/expose-istiod.yaml > expose-istiod.yaml
    cat expose-istiod.yaml
  2. Apply the resources to your workload cluster.

    kubectl apply -n istio-system -f expose-istiod.yaml
  3. For multi-network setups, such as if your VM and cluster are connected to different networks, download the Istio gateway resource to enable AUTO_PASSTHROUGH for cross-network traffic through the east-west gateway.

    curl -0L https://raw.githubusercontent.com/istio/istio/master/samples/multicluster/expose-services.yaml > expose-services.yaml
    cat expose-services.yaml
  4. Apply the resource to your workload cluster.

    kubectl apply -n istio-system -f expose-services.yaml

Step 3: Onboard the VM with meshctl

Now that the Istio control plane and east-west gateway are installed and the control plane is exposed for VM access, use the meshctl vm onboard command to onboard the VM to your Gloo Mesh setup.

meshctl vm onboard --vm-address ${VM_HOSTNAME} --ssh-private-key ${SSH_KEY} [--vm-password ${VM_PASSWORD} --namespace <VM_NS> --service-account <VM_SA> --vm-name <VM_NAME> --cluster-id ${WORKLOAD_CLUSTER1} --output vm-files-output --skip-checks]

Example output after the meshctl command successfully onboards the VM:

πŸ”Ž Performing vm pre-onboard checks...
βœ…  vm pre-onboard checks succeeded!
Warning: a security token for namespace <VM_NS> and service account <VM_SA> has been generated and stored at "vm-files-output/istio-token"

βœ…  Configuration generation into directory output was successful

βœ…  Configuration sent to VM <VM_NAME>

Flags for the meshctl vm onboard command:

Flag Description
vm-address Required: The VM hostname that can be accessed by Gloo Mesh.
ssh-private-key Required for SSH access: The SSH private key file so that Gloo Mesh can authenticate with the VM.
vm-password Optional for SSH access: If you are not logged in as root but have admin access to the VM, you might also need the VM password for sudo access.
namespace Optional: The namespace that hosts the WorkloadGroup and WorkloadEntry resources that are created for the VM. If not provided, a namespace is created called gloo-mesh-vm.
service-account Provide an existing service account, or omit this value so that meshctl creates a service account in the VM namespace for you. Note that if create-sa is set to false, you cannot omit this value, but must provide an existing service account. The service account is used to generate a Kubernetes JSON Web Token (JWT) called istio-token. This JWT is used to get certificates from the Istio certificate authority.
vm-name Optional: A name of the VM to use in the WorkloadGroup and WorkloadEntry resources. If not provided, the vm-address is used.
cluster-id Optional: The name of the cluster as it is registered with Gloo Mesh to onboard the VM to. If not provided, Gloo Mesh attempts to imply the cluster from the Istio injection configmap.
output Optional: Output the files generated by this command to the vm-files-output directory on your local machine. Include this flag if you do not have SSH access to your VM.
skip-checks Optional: Skip the Istio installation checks that ensure istio-eastwestgateway is installed and istiod and the necessary services are exposed.

Onboard without SSH

If you do not have SSH access to your VM, you can manually onboard the VM by using the files generated by the meshctl vm onboard command.

  1. Run the meshctl vm onboard command and include the --output vm-files-output flag.

    meshctl vm onboard --vm-address ${VM_HOSTNAME} [--namespace <VM_NS> --service-account <VM_SA> --vm-name <VM_NAME> --cluster-id ${WORKLOAD_CLUSTER1} --output vm-files-output]

    This command outputs the following files to the vm-files-output directory on your local machine:

    • cluster.env: Contains metadata that identifies which namespace, service account, network CIDR, and (optionally) inbound ports to capture.
    • istio-token: A Kubernetes token used to get certificates from the CA.
    • mesh.yaml: Provides the ProxyConfig to configure the discoveryAddress, health-checking probes, and some authentication options.
    • root-cert.pem: The root certificate used to authenticate.
    • hosts: An addendum to /etc/hosts that the proxy uses to reach istiod for xDS. This is only created if the discovery address is not used directly when --set-hosts=true is set or in Istio version 1.11 or earlier.
  2. Copy the output files to any directory that you want to use, ${VM_DIR}, on the VM.

    scp vm-files-output/* [USERNAME]@[INSTANCE_NAME]:${VM_DIR}
  3. Install the package containing the Istio sidecar on the VM.

       curl -LO https://storage.googleapis.com/istio-release/releases/${ISTIO_IMAGE}/deb/istio-sidecar.deb
       sudo dpkg -i istio-sidecar.deb
       curl -LO https://storage.googleapis.com/istio-release/releases/${ISTIO_IMAGE}/rpm/istio-sidecar.rpm
       sudo rpm -i istio-sidecar.rpm

  4. Copy the Istio token file to the /etc/certs directory.

    sudo  mkdir -p /var/run/secrets/tokens
    sudo cp "${VM_DIR}"/istio-token /var/run/secrets/tokens/istio-token
  5. Copy the root certificate to the /etc/certs directory.

    sudo mkdir -p /etc/certs
    sudo cp "${VM_DIR}"/root-cert.pem /etc/certs/root-cert.pem
  6. Copy the cluster.env to /var/lib/istio/envoy/.

    sudo cp "${VM_DIR}"/cluster.env /var/lib/istio/envoy/cluster.env
  7. Copy the mesh configuration to /etc/istio/config/mesh.

    sudo cp "${VM_DIR}"/mesh.yaml /etc/istio/config/mesh
  8. Add the istiod host to /etc/hosts. Note that if you are not setting the discovery address directly, you must set /etc/hosts.

    sudo sh -c 'cat $(eval echo ~$SUDO_USER)/hosts >> /etc/hosts'
  9. Transfer ownership of the files in /etc/certs/ and /var/lib/istio/envoy/ to istio-proxy.

    sudo mkdir -p /etc/istio/proxy
    sudo chown -R istio-proxy /var/lib/istio /etc/certs /etc/istio/proxy /etc/istio/config /var/run/secrets /etc/certs/root-cert.pem
  10. Start Istio on the VM.

    sudo systemctl start istio

Step 4: Verify that onboarding was successful

Verify that the onboarding process was successful and test the connection between your workload cluster and the VM.

Check onboarding

  1. In your VM, check that Istio was successfully started in your VM.

    systemctl status istio
  2. From your local machine, check that the onboarding process is complete.

    meshctl vm check --vm-address ${VM_HOSTNAME} --ssh-private-key ${SSH_KEY} --output output_logs

    Example output, in which the output directory contains the Istio logs from the Virtual Machine:

    βœ…  ads - caches have been synced
    βœ…  sds - server for workload certificate started
    βœ…  xds - connected to upstream xds server
    Istio logs output to output_logs/${VM_HOSTNAME}-istio.log

Test VM connection to cluster

To test the connection from your VM to the cluster, deploy a sample app and access the app from your VM.

  1. In your cluster, label the sample app namespace, such as default, for Istio sidecar injection.

    kubectl label ns default istio-injection=enabled 
  2. Create the Istio helloworld sample app.

    kubectl apply -f - <<EOF
    apiVersion: v1
    kind: Service
      name: helloworld
        app: helloworld
        service: helloworld
      - port: 5000
        name: http
        app: helloworld

apiVersion: apps/v1 kind: Deployment metadata: name: helloworld-v1 labels: app: helloworld version: v1 spec: replicas: 1 selector: matchLabels: app: helloworld version: v1 template: metadata: labels: app: helloworld version: v1 spec: containers: - name: helloworld image: docker.io/istio/examples-helloworld-v1 resources: requests: cpu: “100m” imagePullPolicy: IfNotPresent #Always ports: - containerPort: 5000

apiVersion: apps/v1 kind: Deployment metadata: name: helloworld-v2 labels: app: helloworld version: v2 spec: replicas: 1 selector: matchLabels: app: helloworld version: v2 template: metadata: labels: app: helloworld version: v2 spec: containers: - name: helloworld image: docker.io/istio/examples-helloworld-v2 resources: requests: cpu: “100m” imagePullPolicy: IfNotPresent #Always ports: - containerPort: 5000 EOF

3. From the VM, curl the `helloworld` service.

curl helloworld.default.svc:5000/hello -v

Example output, after the `helloworld` service is hit from the Virtual Machine:

GET /hello HTTP/1.1 Host: helloworld.default.svc:5000 User-Agent: curl/7.64.0 Accept: / Hello version: v1, instance: helloworld-v1-776f57d5f6-rcrj8

### Test cluster connection to VM

To test the connection from your cluster to the VM, deploy an HTTP server and access the server from your cluster.

1. On the VM, start a simple HTTP server on port 80.

python3 -m http.server 80

2. On your cluster, create a Kubernetes service that represents the HTTP server running on the VM. This adds the HTTP server workload to the Istio service mesh, and creates the Kubernetes service name `simple-server-vm.gloo-mesh-vm` to access the HTTP server on the VM.
kubectl apply -n gloo-mesh-vm -f - <<EOF
apiVersion: v1
kind: Service
  name: simple-server-vm
    app: simple-server-vm
  - port: 80
    name: http-vm
    targetPort: 80
    app: simple-server-vm     
  1. Start a busybox pod in your cluster.

    kubectl run curl --image=radial/busyboxplus:curl -i --tty
  2. In the pod's shell, curl the service name for the HTTP server.

    [ root@curl:/ ]$ curl simple-server-vm.gloo-mesh-vm

    Example output, in which the VM directory listing is included in the response body:

    <!DOCTYPE HTML PUBLIC "-//W3C//DTD HTML 4.01//EN" "http://www.w3.org/TR/html4/strict.dt
    <meta http-equiv="Content-Type" content="text/html; charset=utf-8">
    <title>Directory listing for /</title>
    <h1>Directory listing for /</h1>


If you run into issues while verifying the onboarding process, or if the VM and cluster can't reach each other, check out the following debugging steps.

Check the Envoy config on VM

On the VM, verify that the endpoints from the cluster are being sent to the VM by checking the Envoy cluster configuration.

curl localhost:15000/clusters

If the endpoints are missing, check the Istio logs on the VM to verify that Istio is running, is successfully connected to the xDS server, and is sending and recieving updates.

cat /var/log/istio/istio.log


  1. On the cluster, remove the WorkloadEntry, WorkloadGroups and ServiceAccount for the VM by removing the VM namespace.

    kubectl delete namespace <VM-NS>
  2. Remove Istio from the VM.

    sudo systemctl stop istio
  3. Remove the Istio-sidecar package from the VM.

    sudo dpkg -r istio-sidecar
    dpkg -s istio-sidecar
    sudo rpm -e istio-sidecar

Reference documentation

For more information about the meshctl vm onboard command, see the CLI documentation or run the help flag for a command.

meshctl vm onboard --help