Onboard a virtual machine your Gloo Mesh Enterprise environment by using meshctl.

As you build your Gloo Mesh environment, you might want to add a virtual machine to your setup. For example, you might run an app or service in a VM that you want to add to the Istio service mesh within a cluster in your Gloo Mesh setup. This guide walks you through exposing the Istio service mesh on a cluster to your VM, and using meshctl commands to onboard the VM to your service mesh. For more information, see the high-level architectural overview of Istio’s virtual machine integration.

Before you begin

  1. Install the following CLI tools.

    • istioctl, the Istio command line tool. The resources in the guide use Istio version 1.18.7-patch3.
    • meshctl, the Gloo Mesh command line tool for bootstrapping Gloo Mesh, registering clusters, describing configured resources, and more. This guide requires version 2.0 or later. To quickly install meshctl:
        curl -sL http://run.solo.io/meshctl/install | sh

  2. Install Gloo Mesh Enterprise into a management cluster and register each workload cluster with Gloo Mesh.

  3. Prepare a virtual machine that meets the following requirements.

    • Currently, Debian and CentOS 8 images are supported. The VM in this guide’s examples uses a Debian GNU/Linux 10 (Buster) image and runs in Google Cloud Platform (GCP).
    • Enable the VM for SSH access, copying files, and running Istio. If you do not have SSH permissions set for the VM, after you install Istio, be sure to follow the steps in Onboard without SSH to manually onboard the VM to your Gloo Mesh setup.
    • If you use a VM that runs in GCP, ensure that the “Allow HTTP traffic” option is selected under the Firewall section when you create the VM.

Step 1: Install the Istio control plane and east-west gateway in a workload cluster

Set up the Istio control plane and an east-west gateway in a workload cluster to ensure that it can be accessed by your VM. Note that even if you already installed Istio in a cluster, review these steps to ensure you implement the settings that are required for VM onboarding.

  1. Save the following environment variables for use with Istio installation.

    • For REPO, use a Gloo Istio repo key that you can get by logging in to the Support Center and reviewing the Istio images built by Solo.io support article. For more information, see Get the Gloo Istio version that you want to use.
    • For ISTIO_IMAGE, save the version that you downloaded, such as 1.18.7-patch3, and append the solo tag, which is required to use many enterprise features. You can optionally append other Gloo Istio tags, as described in About Gloo Istio. If you downloaded a different version than the following, make sure to specify that version instead.
    • For REVISION, take the Istio major and minor version numbers and replace the period with a hyphen, such as 1-18.
    • For CLUSTER_NAME, save the name of the workload cluster that you want to install Istio into.
      export REPO=<repo-key>
export ISTIO_IMAGE=1.18.7-patch3-solo
export REVISION=1-18
export CLUSTER_NAME=$REMOTE_CLUSTER1

  2. Install the Istio control plane in a workload cluster that you registered with Gloo Mesh. For example, follow steps 1 and 2 in Deploy Istio in production or follow the steps in Install Istio by using Gloo Mesh.

    • Enable autoregistration in your installation by setting PILOT_ENABLE_WORKLOAD_ENTRY_AUTOREGISTRATION to true. Autoregistration allows the istiod control plane to create the workload entries associated with your VM automatically after the VM is onboarded.
    • You can optionally enable automatic health checks for the VM workload entry by setting PILOT_ENABLE_WORKLOAD_ENTRY_HEALTHCHECKS=true to true.
    • Verify that third-party tokens are enabled in your cluster by following the steps describe in the Istio docs. If third-party tokens are not enabled, you must include the --set values.global.jwtPolicy=third-party-jwt setting.
      istioctl install -f <istiod-kubernetes-values>.yaml --set values.pilot.env.PILOT_ENABLE_WORKLOAD_ENTRY_AUTOREGISTRATION=true [--set values.pilot.env.PILOT_ENABLE_WORKLOAD_ENTRY_HEALTHCHECKS=true --set values.global.jwtPolicy=third-party-jwt]

  3. Prepare an IstioOperator resource for an Istio east-west gateway. The LoadBalancer service that exposes the east-west gateway allows the VM to access the cluster’s service mesh and be onboarded into the mesh. After the VM is onboarded, all traffic sent to and from the VM goes through the east-west gateway.

    • Ensure that the east-west gateway is deployed to the istio-system namespace. If the gateway is deployed to a different namespace, health checks for the VM workload entry will fail.
    • The gateway must be named istio-eastwestgateway, or health checks for the VM workload entry will fail.
    • The gateway must be externally exposed by a LoadBalancer service.
    • Port 15012 must be exposed on the gateway with the name tls-istiod so that the VM can discover the east-west gateway in this cluster.
      cat <<EOF > ./eastwest-gateway.yaml
apiVersion: install.istio.io/v1alpha1
kind: IstioOperator
metadata:
  name: eastwest-gateway-$REVISION
  namespace: istio-system
spec:
  # No control plane components are installed
  profile: empty
  # Solo.io Istio distribution repository; required for Gloo Istio.
  hub: $REPO
  # The Solo.io Gloo Istio version
  tag: $ISTIO_IMAGE
  # Istio revision to create resources with
  revision: $REVISION

  components:
    ingressGateways:
    # Enable the default east-west gateway
      - name: istio-eastwestgateway
        namespace: istio-system
        enabled: true
        label:
          istio: eastwestgateway
          version: $REVISION
          app: istio-eastwestgateway
          # Matches spec.values.global.network in the istiod deployment
          topology.istio.io/network: $CLUSTER_NAME
        k8s:
          hpaSpec:
            maxReplicas: 5
            metrics:
              - resource:
                  name: cpu
                  targetAverageUtilization: 60
                type: Resource
            minReplicas: 2
            scaleTargetRef:
              apiVersion: apps/v1
              kind: Deployment
              name: istio-eastwestgateway
          strategy:
            rollingUpdate:
              maxSurge: 100%
              maxUnavailable: 25%
          resources:
            limits:
              cpu: 2000m
              memory: 1024Mi
            requests:
              cpu: 100m
              memory: 40Mi
          service:
            # You will need the service externally exposed to onboard VMs
            type: LoadBalancer
            ports:
              # Health check port. For AWS ELBs, this port must be listed first.
              - name: status-port
                port: 15021
                targetPort: 15021
              # Port for multicluster mTLS passthrough; required for Gloo Mesh east/west routing
              - port: 15443
                targetPort: 15443
                # Gloo Mesh looks for this default name 'tls' on a gateway
                name: tls
              # Port for VM onboarding
              - port: 15012
                targetPort: 15012
                # Required for VM onboarding discovery address
                name: tls-istiod
          overlays:
            - apiVersion: apps/v1
              kind: Deployment
              name: istio-eastwestgateway
              patches:
                # Sleep 25s on pod shutdown to allow connections to drain
                - path: spec.template.spec.containers.[name:istio-proxy].lifecycle
                  value:
                    preStop:
                      exec:
                        command:
                        - sleep
                        - "25"
                # Schedule pods on separate nodes if possible
                - path: spec.template.spec.affinity
                  value:
                    podAntiAffinity:
                      preferredDuringSchedulingIgnoredDuringExecution:
                      - podAffinityTerm:
                          labelSelector:
                            matchExpressions:
                            - key: app
                              operator: In
                              values:
                              - istio-eastwestgateway
                          topologyKey: kubernetes.io/hostname
                        weight: 100

  # Helm values overrides
  values:
    gateways:
      istio-ingressgateway:
        # Enable gateway injection
        injectionTemplate: gateway
    # https://istio.io/v1.5/docs/reference/config/installation-options/#global-options
    global:
      # Required for connecting VirtualMachines to the mesh
      network: $CLUSTER_NAME
      # Required for annotating Istio metrics with the cluster name.
      # Must match the trustDomain
      multiCluster:
        clusterName: $CLUSTER_NAME
EOF

  4. Update the file with the environment values you set earlier, and install the east-west gateway.

      envsubst < eastwest-gateway.yaml > eastwest-gateway-values.yaml
kubectl apply -f eastwest-gateway-values.yaml

  5. Verify that the east-west gateway load balancer service is assigned an external address.

      kubectl get svc -n istio-system

    Example output:

      NAME                                 TYPE           CLUSTER-IP      EXTERNAL-IP   PORT(S)                                                                                                      AGE
istio-eastwestgateway            LoadBalancer   10.96.166.166   <externalip>  status-port:15021►30075 tls:15443►30110 tls-istiod:15012►30907   13s

Step 2: Expose the Istio control plane to the VM

Create a Gateway and VirtualService so that the VM can reach the istiod control plane on your workload cluster through the east-west gateway.

  1. Download the Istio gateway and virtual service resources that allow passthrough to istiod.

      curl -0L https://raw.githubusercontent.com/istio/istio/master/samples/multicluster/expose-istiod.yaml > expose-istiod.yaml
cat expose-istiod.yaml

  2. Apply the resources to your workload cluster.

      kubectl apply -n istio-system -f expose-istiod.yaml

  3. For multi-network setups, such as if your VM and cluster are connected to different networks, download the Istio gateway resource to enable AUTO_PASSTHROUGH for cross-network traffic through the east-west gateway.

      curl -0L https://raw.githubusercontent.com/istio/istio/master/samples/multicluster/expose-services.yaml > expose-services.yaml
cat expose-services.yaml

  4. Apply the resource to your workload cluster.

      kubectl apply -n istio-system -f expose-services.yaml

Step 3: Onboard the VM with meshctl

Now that the Istio control plane and east-west gateway are installed and the control plane is exposed for VM access, use the meshctl vm onboard command to onboard the VM to your Gloo Mesh setup.

  meshctl vm onboard --vm-address ${VM_HOSTNAME} --ssh-private-key ${SSH_KEY} [--vm-password ${VM_PASSWORD} --namespace <VM_NS> --service-account <VM_SA> --vm-name <VM_NAME> --cluster-id ${WORKLOAD_CLUSTER1} --output vm-files-output --skip-checks]

Example output after the meshctl command successfully onboards the VM:

  🔎 Performing vm pre-onboard checks...
✅  vm pre-onboard checks succeeded!
Warning: a security token for namespace <VM_NS> and service account <VM_SA> has been generated and stored at "vm-files-output/istio-token"

✅  Configuration generation into directory output was successful

✅  Configuration sent to VM <VM_NAME>

Flags for the meshctl vm onboard command:

FlagDescription
vm-addressRequired: The VM hostname that can be accessed by Gloo Mesh.
ssh-private-keyRequired for SSH access: The SSH private key file so that Gloo Mesh can authenticate with the VM.
vm-passwordOptional for SSH access: If you are not logged in as root but have admin access to the VM, you might also need the VM password for sudo access.
namespaceOptional: The namespace that hosts the WorkloadGroup and WorkloadEntry resources that are created for the VM. If not provided, a namespace is created called gloo-mesh-vm.
service-accountProvide an existing service account, or omit this value so that meshctl creates a service account in the VM namespace for you. Note that if create-sa is set to false, you cannot omit this value, but must provide an existing service account. The service account is used to generate a Kubernetes JSON Web Token (JWT) called istio-token. This JWT is used to get certificates from the Istio certificate authority.
vm-nameOptional: A name of the VM to use in the WorkloadGroup and WorkloadEntry resources. If not provided, the vm-address is used.
cluster-idOptional: The name of the cluster as it is registered with Gloo Mesh to onboard the VM to. If not provided, Gloo Mesh attempts to imply the cluster from the Istio injection configmap.
outputOptional: Output the files generated by this command to the vm-files-output directory on your local machine. Include this flag if you do not have SSH access to your VM.
skip-checksOptional: Skip the Istio installation checks that ensure istio-eastwestgateway is installed and istiod and the necessary services are exposed.

Onboard without SSH

If you do not have SSH access to your VM, you can manually onboard the VM by using the files generated by the meshctl vm onboard command.

  1. Run the meshctl vm onboard command and include the --output vm-files-output flag.

      meshctl vm onboard --vm-address ${VM_HOSTNAME} [--namespace <VM_NS> --service-account <VM_SA> --vm-name <VM_NAME> --cluster-id ${WORKLOAD_CLUSTER1} --output vm-files-output]

    This command outputs the following files to the vm-files-output directory on your local machine:

    • cluster.env: Contains metadata that identifies which namespace, service account, network CIDR, and (optionally) inbound ports to capture.
    • istio-token: A Kubernetes token used to get certificates from the CA.
    • mesh.yaml: Provides the ProxyConfig to configure the discoveryAddress, health-checking probes, and some authentication options.
    • root-cert.pem: The root certificate used to authenticate.
    • hosts: An addendum to /etc/hosts that the proxy uses to reach istiod for xDS. This is only created if the discovery address is not used directly when --set-hosts=true is set or in Istio version 1.11 or earlier.

  2. Copy the output files to any directory that you want to use, ${VM_DIR}, on the VM.

      scp vm-files-output/* [USERNAME]@[INSTANCE_NAME]:${VM_DIR}

  3. Install the package containing the Istio sidecar on the VM.

      curl -LO https://storage.googleapis.com/istio-release/releases/${ISTIO_IMAGE}/deb/istio-sidecar.deb
sudo dpkg -i istio-sidecar.deb
  
      curl -LO https://storage.googleapis.com/istio-release/releases/${ISTIO_IMAGE}/rpm/istio-sidecar.rpm
sudo rpm -i istio-sidecar.rpm

  4. Copy the Istio token file to the /etc/certs directory.

      sudo  mkdir -p /var/run/secrets/tokens
sudo cp "${VM_DIR}"/istio-token /var/run/secrets/tokens/istio-token

  5. Copy the root certificate to the /etc/certs directory.

      sudo mkdir -p /etc/certs
sudo cp "${VM_DIR}"/root-cert.pem /etc/certs/root-cert.pem

  6. Copy the cluster.env to /var/lib/istio/envoy/.

      sudo cp "${VM_DIR}"/cluster.env /var/lib/istio/envoy/cluster.env

  7. Copy the mesh configuration to /etc/istio/config/mesh.

      sudo cp "${VM_DIR}"/mesh.yaml /etc/istio/config/mesh

  8. Add the istiod host to /etc/hosts. Note that if you are not setting the discovery address directly, you must set /etc/hosts.

      sudo sh -c 'cat $(eval echo ~$SUDO_USER)/hosts >> /etc/hosts'

  9. Transfer ownership of the files in /etc/certs/ and /var/lib/istio/envoy/ to istio-proxy.

      sudo mkdir -p /etc/istio/proxy
sudo chown -R istio-proxy /var/lib/istio /etc/certs /etc/istio/proxy /etc/istio/config /var/run/secrets /etc/certs/root-cert.pem

  10. Start Istio on the VM.

      sudo systemctl start istio

Step 4: Verify that onboarding was successful

Verify that the onboarding process was successful and test the connection between your workload cluster and the VM.

Check onboarding

  1. In your VM, check that Istio was successfully started in your VM.

      systemctl status istio

  2. From your local machine, check that the onboarding process is complete.

      meshctl vm check --vm-address ${VM_HOSTNAME} --ssh-private-key ${SSH_KEY} --output output_logs

    Example output, in which the output directory contains the Istio logs from the Virtual Machine:

      ✅  ads - caches have been synced
✅  sds - server for workload certificate started
✅  xds - connected to upstream xds server
Istio logs output to output_logs/${VM_HOSTNAME}-istio.log

Test VM connection to cluster

To test the connection from your VM to the cluster, deploy a sample app and access the app from your VM.

  1. In your cluster, label the sample app namespace, such as default, for Istio sidecar injection.

      kubectl label ns default istio-injection=enabled

  2. Create the Istio helloworld sample app.

      kubectl apply -f https://raw.githubusercontent.com/istio/istio/master/samples/helloworld/helloworld.yaml

  3. From the VM, curl the helloworld service.

      curl helloworld.default.svc:5000/hello -v

    Example output, after the helloworld service is hit from the Virtual Machine:

      *   Trying <EAST-WEST-GATEWAY-IP>...
* TCP_NODELAY set
* Expire in 200 ms for 4 (transfer 0x557d593a3fb0)
* Connected to helloworld.default.svc (<EAST-WEST-GATEWAY-IP>) port 5000 (#0)
> GET /hello HTTP/1.1
> Host: helloworld.default.svc:5000
> User-Agent: curl/7.64.0
> Accept: */*
Hello version: v1, instance: helloworld-v1-776f57d5f6-rcrj8

Test cluster connection to VM

To test the connection from your cluster to the VM, deploy an HTTP server and access the server from your cluster.

  1. On the VM, start a simple HTTP server on port 80.

      python3 -m http.server 80

  2. On your cluster, create a Kubernetes service that represents the HTTP server running on the VM. This adds the HTTP server workload to the Istio service mesh, and creates the Kubernetes service name simple-server-vm.gloo-mesh-vm to access the HTTP server on the VM.

      kubectl apply -n gloo-mesh-vm -f - <<EOF
apiVersion: v1
kind: Service
metadata:
  name: simple-server-vm
  labels:
    app: simple-server-vm
spec:
  ports:
  - port: 80
    name: http-vm
    targetPort: 80
  selector:
    app: simple-server-vm     
EOF

  3. Start a busybox pod in your cluster.

      kubectl run curl --image=radial/busyboxplus:curl -i --tty

  4. In the pod’s shell, curl the service name for the HTTP server.

      [ root@curl:/ ]$ curl simple-server-vm.gloo-mesh-vm

    Example output, in which the VM directory listing is included in the response body:

      <!DOCTYPE HTML PUBLIC "-//W3C//DTD HTML 4.01//EN" "http://www.w3.org/TR/html4/strict.dt
d">
<html>
<head>
<meta http-equiv="Content-Type" content="text/html; charset=utf-8">
<title>Directory listing for /</title>
</head>
<body>
<h1>Directory listing for /</h1>
<hr>
...

Debugging

If you run into issues while verifying the onboarding process, or if the VM and cluster can’t reach each other, check out the following debugging steps.

Check the Envoy config on VM

On the VM, verify that the endpoints from the cluster are being sent to the VM by checking the Envoy cluster configuration.

  curl localhost:15000/clusters

If the endpoints are missing, check the Istio logs on the VM to verify that Istio is running, is successfully connected to the xDS server, and is sending and recieving updates.

  cat /var/log/istio/istio.log

Uninstall

  1. On the cluster, remove the WorkloadEntry, WorkloadGroups and ServiceAccount for the VM by removing the VM namespace.

      kubectl delete namespace <VM-NS>

  2. Remove Istio from the VM.

      sudo systemctl stop istio

  3. Remove the Istio-sidecar package from the VM.

      sudo dpkg -r istio-sidecar
dpkg -s istio-sidecar
  
      sudo rpm -e istio-sidecar

Reference documentation

For more information about the meshctl vm onboard command, see the CLI documentation or run the help flag for a command.

  meshctl vm onboard --help