Integrate with Vault

Vault is a popular open source secret management tool that you can use to set up a secure, private key infrastructure (PKI) and manage TLS certificates. In this setup, you install a Vault instance in the Gloo management cluster that serves as the root certificate authority. The root CA certificate and private key that are stored in Vault are used to sign and issue Istio intermediate CA certificates.

To enable Gloo Mesh to automatically derive intermediate CA certificates from the root CA in Vault, istiod is injected with the istiod-agent sidecar that authenticates and sends certificate signing requests to Vault. Instead of writing the intermediate CA certificate and key to the cacerts Kubernetes secret, the istiod-agent stores the credentials in memory, which adds an additional security layer to this approach. When the istiod-agent sidecar is deleted, the private key is also deleted. Istiod reads the intermediate CA key from the istiod-agent memory directly when signing leaf certificates for the workloads in the service mesh.

For more information about this approach, see Option 4: Integrate with Vault.

The instructions in this guide are also provided as an example script that you can review and adapt for your own use. Note that this script was written for Kubernetes version 1.21 and might need to be adjusted for other Kubernetes versions.

Before you begin

This guide assumes that you use the same names for components like clusters, workspaces, and namespaces as in the getting started. If you have different names, make sure to update the sample configuration files in this guide.
  1. Complete the multicluster getting started guide to set up the following testing environment.
    • Three clusters along with environment variables for the clusters and their Kubernetes contexts.
    • The Gloo Platform CLI, meshctl, along with other CLI tools such as kubectl and istioctl.
    • The Gloo management server in the management cluster, and the Gloo agents in the workload clusters.
    • Istio installed in the workload clusters.
    • A simple Gloo workspace setup.
  2. Install Bookinfo and other sample apps.
  3. The default openssl version that is included in macOS is LibreSSL, which does not work with these instructions.

    Make sure that you have the OpenSSL version of openssl, not LibreSSL. The openssl version must be at least 1.1.

    1. Check the openssl version that is installed. If you see LibreSSL in the output, continue to the next step.
      openssl version
      
    2. Install the OpenSSL version (not LibreSSL). For example, you might use Homebrew.
      brew install openssl
      
    3. Review the output of the OpenSSL installation for the path of the binary file. You can choose to export the binary to your path, or call the entire path whenever the following steps use an openssl command.
      • For example, openssl might be installed along the following path: /usr/local/opt/openssl@3/bin/
      • To run commands, you can append the path so that your terminal uses this installed version of OpenSSL, and not the default LibreSSL. /usr/local/opt/openssl@3/bin/openssl req -new -newkey rsa:4096 -x509 -sha256 -days 3650...
  4. Save the kubeconfig contexts for your clusters. Run kubectl config get-contexts, look for your cluster in the CLUSTER column, and get the context name in the NAME column. Note: Do not use context names with underscores. The context name is used as a SAN specification in the generated certificate that connects workload clusters to the management cluster, and underscores in SAN are not FQDN compliant. You can rename a context by running kubectl config rename-context "<oldcontext>" <newcontext>.
    export MGMT_CLUSTER=<mgmt-cluster-name>
    export REMOTE_CLUSTER=<remote-cluster-name>
    export MGMT_CONTEXT=<management-cluster-context>
    export REMOTE_CONTEXT=<remote-cluster-context>
    

Install and set up Vault

If you already have Vault installed, you can use your existing deployment.
  1. If not added already, add the HashiCorp Helm repository to your management cluster.

    helm repo add hashicorp https://helm.releases.hashicorp.com --kube-context ${MGMT_CONTEXT}
    helm repo update
    
  2. Generate a root CA certificate and key for Vault. You can update the subj field to your domain.

    openssl req -new -newkey rsa:4096 -x509 -sha256 \
        -days 3650 -nodes -out root-cert.pem -keyout root-key.pem \
        -subj "/O=solo.io"
    
  3. In the management cluster, install Vault in dev mode and enable debugging logs. For more information about setting up Vault in Kubernetes, see the Vault docs.

    helm install -n vault vault hashicorp/vault --set "injector.enabled=false" --set "server.logLevel=debug" --set "server.dev.enabled=true" --set "server.service.type=LoadBalancer" --kube-context="${MGMT_CONTEXT}" --create-namespace
    kubectl --context $MGMT_CONTEXT wait --for condition=Ready -n vault pod/vault-0
    

    Example output:

    pod/vault-0 condition met
    
  4. Enable Vault userpass.

    kubectl --context="${MGMT_CONTEXT}" exec -n vault vault-0 -- /bin/sh -c 'vault auth enable userpass'
    kubectl --context="${MGMT_CONTEXT}" exec -n vault vault-0 -- /bin/sh -c 'vault write auth/userpass/users/admin password=admin policies=admins'
    

    Example output:

    Success! Enabled userpass auth method at: userpass/
    Success! Data written to: auth/userpass/users/admin
    
  5. Enable Vault authentication along a path for the workload cluster.

    kubectl --context="${MGMT_CONTEXT}" exec -n vault vault-0 -- /bin/sh -c "vault auth enable -path=kube-${REMOTE_CLUSTER1}-mesh-auth kubernetes"
    kubectl --context="${MGMT_CONTEXT}" exec -n vault vault-0 -- /bin/sh -c "vault auth enable -path=kube-${REMOTE_CLUSTER2}-mesh-auth kubernetes"
    

    Example output:

    Success! Enabled kubernetes auth method at: kube-cluster1-mesh-auth/
    
  6. Get the API token for the istiod-service-account. Note that depending on your Kubernetes version, the token is automatically created for you or must be created manually.

    1. Get the secret that the istiod-service-account uses.
      VAULT_SA_NAME_C1=$(kubectl --context $REMOTE_CONTEXT1 get sa istiod-service-account -n istio-system -o jsonpath="{.secrets[*]['name']}")
      VAULT_SA_NAME_C2=$(kubectl --context $REMOTE_CONTEXT2 get sa istiod-service-account -n istio-system -o jsonpath="{.secrets[*]['name']}")
      
    2. Get the token values and save it as an environment variable.
      SA_TOKEN_C1=$(kubectl --context $REMOTE_CONTEXT1 get secret $VAULT_SA_NAME_C1 -n istio-system -o 'go-template={{ .data.token }}' | base64 --decode)
      SA_TOKEN_C2=$(kubectl --context $REMOTE_CONTEXT2 get secret $VAULT_SA_NAME_C2 -n istio-system -o 'go-template={{ .data.token }}' | base64 --decode)
      echo $SA_TOKEN_C1
      echo $SA_TOKEN_C2
      
    1. Create a token for the istiod-service-account service account and store this token in a secret.

      kubectl apply --context $REMOTE_CONTEXT1 -f - <<EOF 
      apiVersion: v1
      kind: Secret
      metadata:
        name: istiod-sa
        namespace: istio-system
        annotations:
          kubernetes.io/service-account.name: istiod-service-account
      type: kubernetes.io/service-account-token
      EOF
      
      kubectl apply --context $REMOTE_CONTEXT2 -f - <<EOF 
      apiVersion: v1
      kind: Secret
      metadata:
        name: istiod-sa
        namespace: istio-system
        annotations:
          kubernetes.io/service-account.name: istiod-service-account
      type: kubernetes.io/service-account-token
      EOF
      
    2. Verify that the token was created.

      kubectl get secret/istiod-sa --context $REMOTE_CONTEXT1 -n istio-system -o yaml 
      kubectl get secret/istiod-sa --context $REMOTE_CONTEXT2 -n istio-system -o yaml 
      
    3. Save the token in an environment variable.

      SA_TOKEN_C1=$(kubectl --context $REMOTE_CONTEXT1 get secret istiod-sa -n istio-system -o 'go-template={{ .data.token }}' | base64 --decode)
      SA_TOKEN_C2=$(kubectl --context $REMOTE_CONTEXT2 get secret istiod-sa -n istio-system -o 'go-template={{ .data.token }}' | base64 --decode)
      echo $SA_TOKEN_C1
      echo $SA_TOKEN_C2
      

      Example output:

      eyJh...
      eyJi...
      

  7. Get the CA certificate for the service account.

    SA_CA_CRT_C1=$(kubectl config view --raw -o json | jq -r --arg wc $REMOTE_CONTEXT1 '. as $c | $c.contexts[] | select(.name == $wc) as $context | $c.clusters[] | select(.name == $context.context.cluster) | .cluster."certificate-authority-data"'| base64 -d)
    SA_CA_CRT_C2=$(kubectl config view --raw -o json | jq -r --arg wc $REMOTE_CONTEXT2 '. as $c | $c.contexts[] | select(.name == $wc) as $context | $c.clusters[] | select(.name == $context.context.cluster) | .cluster."certificate-authority-data"'| base64 -d)
    echo $SA_CA_CRT_C1
    echo $SA_CA_CRT_C2
    

    Example output:

    -----BEGIN CERTIFICATE-----
    ...
    -----END CERTIFICATE-----
    -----BEGIN CERTIFICATE-----
    ...
    -----END CERTIFICATE-----
    
  8. Get the address of your cluster.

    K8S_ADDR_C1=$(kubectl config view -o json | jq -r --arg wc $REMOTE_CONTEXT1 '. as $c | $c.contexts[] | select(.name == $wc) as $context | $c.clusters[] | select(.name == $context.context.cluster) | .cluster.server ')
    K8S_ADDR_C2=$(kubectl config view -o json | jq -r --arg wc $REMOTE_CONTEXT2 '. as $c | $c.contexts[] | select(.name == $wc) as $context | $c.clusters[] | select(.name == $context.context.cluster) | .cluster.server ')
    echo $K8S_ADDR_C1
    echo $K8S_ADDR_C2
    

    Example output:

    https://34.xxx.xxx.xxx
    https://35.xxx.xxx.xxx
    
  9. Set the Kubernetes auth config for Vault to the mounted service account token.

    kubectl --context="${MGMT_CONTEXT}" exec -n vault vault-0 -- /bin/sh -c "vault write auth/kube-${REMOTE_CLUSTER1}-mesh-auth/config \
    token_reviewer_jwt="$SA_TOKEN_C1" \
    kubernetes_host="$K8S_ADDR_C1" \
    kubernetes_ca_cert='$SA_CA_CRT_C1' \
    disable_local_ca_jwt="true" \
    issuer='https://kubernetes.default.svc.cluster.local'"
    
    kubectl --context="${MGMT_CONTEXT}" exec -n vault vault-0 -- /bin/sh -c "vault write auth/kube-${REMOTE_CLUSTER2}-mesh-auth/config \
    token_reviewer_jwt="$SA_TOKEN_C2" \
    kubernetes_host="$K8S_ADDR_C2" \
    kubernetes_ca_cert='$SA_CA_CRT_C2' \
    disable_local_ca_jwt="true" \
    issuer='https://kubernetes.default.svc.cluster.local'"
    

    Example output:

    Success! Data written to: auth/kube-${REMOTE_CLUSTER}-mesh-auth/config
    Success! Data written to: auth/kube-${REMOTE_CLUSTER}-mesh-auth/config
    
  10. Bind the istiod service account to the Vault PKI policy.

    kubectl --context="${MGMT_CONTEXT}" exec -n vault vault-0 -- /bin/sh -c "vault write \
    auth/kube-${REMOTE_CLUSTER1}-mesh-auth/role/gen-int-ca-istio-${REMOTE_CLUSTER1}-mesh \
    bound_service_account_names=istiod-service-account \
    bound_service_account_namespaces=istio-system \
    policies=gen-int-ca-istio-${REMOTE_CLUSTER1}-mesh \
    ttl=720h"
    
    kubectl --context="${MGMT_CONTEXT}" exec -n vault vault-0 -- /bin/sh -c "vault write \
    auth/kube-${REMOTE_CLUSTER2}-mesh-auth/role/gen-int-ca-istio-${REMOTE_CLUSTER2}-mesh \
    bound_service_account_names=istiod-service-account \
    bound_service_account_namespaces=istio-system \
    policies=gen-int-ca-istio-${REMOTE_CLUSTER2}-mesh \
    ttl=720h"
    

    Example output:

    Success! Data written to: auth/kube-${REMOTE_CLUSTER1}-mesh-auth/role/gen-int-ca-istio-${REMOTE_CLUSTER1}-mesh
    Success! Data written to: auth/kube-${REMOTE_CLUSTER2}-mesh-auth/role/gen-int-ca-istio-${REMOTE_CLUSTER2}-mesh
    
  11. Initialize the Vault PKI.

    kubectl --context="${MGMT_CONTEXT}" exec -n vault vault-0 -- /bin/sh -c 'vault secrets enable pki'
    

    Example output:

    Success! Enabled the pki secrets engine at: pki/
    
  12. Set the Vault CA to the pem_bundle.

    kubectl --context="${MGMT_CONTEXT}" exec -n vault vault-0 -- /bin/sh -c "vault write -format=json pki/config/ca pem_bundle=\"$(cat root-key.pem root-cert.pem)\""
    

    Example output:

    {
      "request_id": "2aa29fd6-9fa3-3edd-2f8b-2a0e4c007e8c",
      "lease_id": "",
      "lease_duration": 0,
      "renewable": false,
      "data": {
        "imported_issuers": null,
        "imported_keys": null,
        "mapping": {
          "aa877391-b4f2-045d-63da-33521c91dc68": "8257875c-4016-f28e-288b-ecca33065097"
        }
      },
      "warnings": null
    }
    
  13. Enable the Vault intermediate cert path. Replace ${REMOTE_CLUSTER} with your cluster's name.

    kubectl --context="${MGMT_CONTEXT}" exec -n vault vault-0 -- /bin/sh -c "vault secrets enable -path=pki_int_${REMOTE_CLUSTER1} pki"
    kubectl --context="${MGMT_CONTEXT}" exec -n vault vault-0 -- /bin/sh -c "vault secrets enable -path=pki_int_${REMOTE_CLUSTER2} pki"
    

    Example output:

    Success! Enabled the pki secrets engine at: pki_int_${REMOTE_CLUSTER1}/
    Success! Enabled the pki secrets engine at: pki_int_${REMOTE_CLUSTER2}/
    
  14. Set the policy for the intermediate cert path. Replace $REMOTE_CLUSTER1 and $REMOTE_CLUSTER2 with your cluster names.

    kubectl --context="${MGMT_CONTEXT}" exec -n vault vault-0 -- /bin/sh -c 'vault policy write gen-int-ca-istio-${REMOTE_CLUSTER1}-mesh - <<EOF
    path "pki_int_${REMOTE_CLUSTER1}/*" {
    capabilities = ["create", "read", "update", "delete", "list"]
    }
    path "pki/cert/ca" {
    capabilities = ["read"]
    }
    path "pki/root/sign-intermediate" {
    capabilities = ["create", "read", "update", "list"]
    }
    EOF'
    
    kubectl --context="${MGMT_CONTEXT}" exec -n vault vault-0 -- /bin/sh -c 'vault policy write gen-int-ca-istio-${REMOTE_CLUSTER2}-mesh - <<EOF
    path "pki_int_${REMOTE_CLUSTER2}/*" {
    capabilities = ["create", "read", "update", "delete", "list"]
    }
    path "pki/cert/ca" {
    capabilities = ["read"]
    }
    path "pki/root/sign-intermediate" {
    capabilities = ["create", "read", "update", "list"]
    }
    EOF'
    

    Example output:

    Success! Uploaded policy: gen-int-ca-istio-${REMOTE_CLUSTER1}-mesh
    Success! Uploaded policy: gen-int-ca-istio-${REMOTE_CLUSTER2}-mesh
    

Now that Vault is set up in your clusters, you can use Vault as an intermediate CA provider. If you see any errors, review the troubleshooting section.

Update Gloo RBAC permissions

The istio-agent sidecar in each cluster needs to read and modify Gloo resources. To enable the necessary RBAC permissions, update the gloo-agent Helm release. You can update the Helm release by adding the following snippet to the YAML configuration file in your GitOps pipeline or directly with the helm upgrade command.

  1. Follow the steps to Get your Helm chart values for the Gloo agent deployment.

  2. Add the following code to your Helm values file.

    istiodSidecar:
        createRoleBinding: true
        istiodServiceAccount:
            name: istiod-1-16-vault
            namespace: istio-system
    
    istiodSidecar:
        createRoleBinding: true
        istiodServiceAccount:
            name: istiod
            namespace: istio-system
    

  3. Set the Gloo version as an environment variable.

    export GLOO_VERSION=2.4.2
    
  4. Make sure that you have the Helm repo for the Gloo agent. Note that you might have a different name for the Helm repo, such as gloo-mesh-agent.

    helm repo add gloo-agent https://storage.googleapis.com/gloo-mesh-enterprise/gloo-mesh-agent --kube-context ${REMOTE_CONTEXT1}
    helm repo update --kube-context ${REMOTE_CONTEXT1}
    
  5. Upgrade the Gloo agent Helm chart with the required RBAC permissions. Note that you might have a different name for the Helm repo, such as gloo-mesh-agent.

    helm upgrade -n gloo-mesh gloo-agent gloo-mesh-agent/gloo-mesh-agent --kube-context="${REMOTE_CONTEXT1}" --version=$GLOO_VERSION -f values-data-plane-env.yaml
    
  6. Repeat these steps for each workload cluster.

Modify istiod

So far, you set up the Gloo agent on each cluster to use Vault to obtain the intermediate CA. Now, you can modify your Istio installation to support fetching and dynamically reloading the intermediate CA from Vault.

These steps vary based on your Istio installation method.

If you installed Istio by using the Istio Lifecycle Manager, follow these steps to update your istiod deployments.

  1. Get the names of your managed Istio installation resource.

    kubectl get IstioLifecycleManager -n gloo-mesh --context $MGMT_CONTEXT
    
  2. Edit the IstioLifecycleManager resource for the istiod control plane to add another installation entry for a canary revision. This canary revision contains the following pilot overlay. Replace $MGMT_PLANE_VERSION with the version that you got in the previous step.

    kubectl edit IstioLifecycleManager -n gloo-mesh --context $MGMT_CONTEXT <installation-name>
    

    For example, add the following 1-18-2-vault canary revision as a new entry in the installations section, which contains the pilot overlay:

    apiVersion: admin.gloo.solo.io/v2
    kind: IstioLifecycleManager
    metadata:
      name: istiod-control-plane
      namespace: gloo-mesh
    spec:
      installations:
        # Existing revision
        - revision: 1-18-2
          clusters:
          - name: cluster1
            # Keep this field set to TRUE so that only the existing revision continues to run
            defaultRevision: true
          - name: cluster2
            defaultRevision: true
          istioOperatorSpec:
            profile: minimal
            hub: $REPO
            tag: $ISTIO_IMAGE
            namespace: istio-system
            ...
        # Canary revision
        - revision: 1-18-2-vault
          clusters:
          - name: cluster1
            # Set this field to FALSE so that only the existing revision continues to run
            defaultRevision: false
          - name: cluster2
            defaultRevision: false
          istioOperatorSpec:
            profile: minimal
            ...
            components:
              pilot:
                k8s:
                  overlays:
                  - apiVersion: apps/v1
                    kind: Deployment
                    name: istiod
                    patches:
                    # override istiod cacerts volume
                    - path: spec.template.spec.volumes[name:cacerts]
                      value: 
                        name: cacerts
                        secret: null
                        emptyDir:
                          medium: Memory
                    # override istiod istiod-agent container to use Solo istiod-agent build
                    - path: spec.template.spec.containers[1]
                      value: 
                        name: istiod-agent
                        image: gcr.io/gloo-mesh/gloo-mesh-istiod-agent:$MGMT_PLANE_VERSION
                        imagePullPolicy: IfNotPresent
                        volumeMounts:
                        - mountPath: /etc/cacerts
                          name: cacerts
                        args: 
                        - sidecar
                        env:
                        - name: PILOT_CERT_PROVIDER
                          value: istiod
                        - name: POD_NAME
                          valueFrom:
                            fieldRef:
                              apiVersion: v1
                              fieldPath: metadata.name
                        - name: POD_NAMESPACE
                          valueFrom:
                            fieldRef:
                              apiVersion: v1
                              fieldPath: metadata.namespace
                        - name: SERVICE_ACCOUNT
                          valueFrom:
                            fieldRef:
                              apiVersion: v1
                              fieldPath: spec.serviceAccountName
                    # override istiod istiod-agent-init init-container to use Solo istiod-agent-init build
                    - path: spec.template.spec.initContainers
                      value: 
                      - name: istiod-agent-init
                        image: gcr.io/gloo-mesh/gloo-mesh-istiod-agent:$MGMT_PLANE_VERSION
                        imagePullPolicy: IfNotPresent
                        volumeMounts:
                        - mountPath: /etc/cacerts
                          name: cacerts
                        args: 
                        - init-container
                        env:
                        - name: PILOT_CERT_PROVIDER
                          value: istiod
                        - name: POD_NAME
                          valueFrom:
                            fieldRef:
                              apiVersion: v1
                              fieldPath: metadata.name
                        - name: POD_NAMESPACE
                          valueFrom:
                            fieldRef:
                              apiVersion: v1
                              fieldPath: metadata.namespace
                        - name: SERVICE_ACCOUNT
                          valueFrom:
                            fieldRef:
                              apiVersion: v1
                              fieldPath: spec.serviceAccountName 
    
  3. Save and close the file. The canary control plane with the pilot overlay is automatically deployed to your cluster.

  4. Verify that the canary istiod control plane resources are deployed alongside your existing control plane in your workload clusters.

    kubectl get all -n istio-system --context $REMOTE_CONTEXT1
    
  5. Switch to the new istiod control plane revision by changing defaultRevision to false for the old revision and to true for the new revision.

    kubectl edit IstioLifecycleManager -n gloo-mesh --context $MGMT_CONTEXT istiod-control-plane
    

    Example:

    apiVersion: admin.gloo.solo.io/v2
    kind: IstioLifecycleManager
    metadata:
      name: istiod-control-plane
      namespace: gloo-mesh
    spec:
      installations:
        # Old revision
        - revision: 1-18-2
          clusters:
          - name: cluster1
            # Set this field to FALSE
            defaultRevision: false
          - name: cluster2
            defaultRevision: false
          istioOperatorSpec:
            profile: minimal
            ...
        # New revision
        - revision: 1-18-2-vault
          clusters:
          - name: cluster1
            # Set this field to TRUE
            defaultRevision: true
          - name: cluster2
            defaultRevision: true
          istioOperatorSpec:
            profile: minimal
            ...
    
  6. To uninstall the previous installation, or if you need to uninstall the canary installation, you can edit the IstioLifecycleManager to remove the revision's entry from the installations list.

If you manually deployed Istio by using the community Istio Helm charts, follow these steps to update your istiod deployments.

These steps use the post rendering feature supported in Helm 3.1 and later, along with Kustomize. These features patch the rendered manifest of the istiod deployment before it is applied to the cluster without the need to modify the Helm chart itself, because deployment patches cannot be added to the istiod Helm chart directly.

  1. Get the version of the Gloo management server that runs in your management cluster.

    export MGMT_PLANE_VERSION=$(meshctl version --kubecontext $MGMT_CONTEXT | jq '.server[].components[] | select(.componentName == "gloo-mesh-mgmt-server") | .images[] | select(.name == "gloo-mesh-mgmt-server") | .version')
    echo $MGMT_PLANE_VERSION
    

    Example output:

    "2.4.2"
    
  2. Save the following patch, which updates the istiod deployment with the gloo-mesh-istiod-agent sidecar to load and store the Vault certificates before the deployment is applied to the cluster.

    cat > vault-patch.yaml <<EOF
    apiVersion: apps/v1
    kind: Deployment
    metadata:
      name: istiod
    spec:
      template:
        spec:
          volumes:
          - name: cacerts
            secret: null
            emptyDir:
              medium: Memory
          # override the istiod-agent container to use the Solo istiod-agent build
          containers:
          - name: istiod-agent
            image: gcr.io/gloo-mesh/gloo-mesh-istiod-agent:$MGMT_PLANE_VERSION
            imagePullPolicy: IfNotPresent
            volumeMounts:
            - mountPath: /etc/cacerts
              name: cacerts
            args: 
            - sidecar
            env:
            - name: PILOT_CERT_PROVIDER
              value: istiod
            - name: POD_NAME
              valueFrom:
                fieldRef:
                  apiVersion: v1
                  fieldPath: metadata.name
            - name: POD_NAMESPACE
              valueFrom:
                fieldRef:
                  apiVersion: v1
                  fieldPath: metadata.namespace
            - name: SERVICE_ACCOUNT
              valueFrom:
                fieldRef:
                  apiVersion: v1
                  fieldPath: spec.serviceAccountName
          # override the istiod-agent-init init-container to use the Solo istiod-agent-init build
          initContainers:
          - name: istiod-agent-init
            image: gcr.io/gloo-mesh/gloo-mesh-istiod-agent:$MGMT_PLANE_VERSION
            imagePullPolicy: IfNotPresent
            volumeMounts:
            - mountPath: /etc/cacerts
              name: cacerts
            args: 
            - init-container
            env:
            - name: PILOT_CERT_PROVIDER
              value: istiod
            - name: POD_NAME
              valueFrom:
                fieldRef:
                  apiVersion: v1
                  fieldPath: metadata.name
            - name: POD_NAMESPACE
              valueFrom:
                fieldRef:
                  apiVersion: v1
                  fieldPath: metadata.namespace
            - name: SERVICE_ACCOUNT
              valueFrom:
                fieldRef:
                  apiVersion: v1
                  fieldPath: spec.serviceAccountName
    EOF
    
  3. To use Kustomize files with Helm rendering, which requires stdin/stdout, create a shell script named kustomize.sh.

    cat > kustomize.sh <<EOF
    #!/bin/sh
    cat > base.yaml
    exec kubectl kustomize
    EOF
    chmod +x ./kustomize.sh
    
  4. Create the kustomization.yaml file, which provides the patch input for Kustomize.

    cat > kustomization.yaml <<EOF
    resources:
    - base.yaml
    patchesStrategicMerge:
    - vault-patch.yaml
    EOF
    
  5. Follow the guides to install or upgrade Istio in your workload clusters. When you run the helm commands in either guide, be sure to include the --post-renderer ./kustomize.sh flag, such as the following:

    helm upgrade --install istiod-${REVISION} istio/istiod \
      --version ${ISTIO_IMAGE} \
      --namespace istio-system \
      --create-namespace \
      --post-renderer ./kustomize.sh \
      --wait \
      -f istiod-kubernetes-values.yaml
    
  6. Repeat the previous step for each workload cluster with Istio installed.

Enable Vault as an intermediate CA provider

Now, federate the two meshes together by using Gloo with Vault to establish trusted communication across the service meshes.

  1. Get the endpoint for the Vault service in the management cluster.
    export VAULT_ENDPOINT="http://$(kubectl get svc/vault -n vault -o wide --context $MGMT_CONTEXT -o jsonpath='{.status.loadBalancer.ingress[0].*}')"
    echo $VAULT_ENDPOINT
    

    Example output:

    http://35.xxx.xxx.xxx
    
  2. Get the name of your Istio mesh.
    export MESH=$(kubectl get meshes -n gloo-mesh --context $REMOTE_CONTEXT -o jsonpath='{.items[*].metadata.name}')
    echo $MESH
    
  3. Create a root trust policy for the workload cluster so that the istiod agent on the workload cluster knows how to communicate with Vault on the management cluster. For more information about root trust policies, see the API docs.
    kubectl apply --context ${REMOTE_CONTEXT} -f - << EOF
    apiVersion: admin.gloo.solo.io/v2
    kind: RootTrustPolicy
    metadata:
      name: ${REMOTE_CLUSTER}
      namespace: gloo-mesh
    spec:
      applyToMeshes:
      - istio:
          clusterSelector:
            mesh: ${MESH}
          namespace: istio-system
          selector:
            app: istiod
            vault: ${REMOTE_CLUSTER}
      config:
        agentCa:
          vault:
            caPath: pki/root/sign-intermediate
            csrPath: pki_int_${REMOTE_CLUSTER}/intermediate/generate/exported
            server: $VAULT_ENDPOINT:8200
            kubernetesAuth:
              mountPath: /v1/auth/kube-${REMOTE_CLUSTER}-mesh-auth
              role: gen-int-ca-istio-${REMOTE_CLUSTER}-mesh
    EOF
    
  4. Restart the istiod deployment. Note that you cannot update Istio resources until istiod is running again.
    kubectl rollout restart deployment -l app=istiod -n istio-system --context ${REMOTE_CONTEXT}
    
  5. Repeat the previous steps for each workload cluster with Istio.

Verify traffic uses the root CA

Now that the Istio control plane is patched with the gloo-mesh-istiod-agent sidecar, you can verify that all of the service mesh traffic is secured by using the root CA that you generated for Vault in the previous section.

To verify, you can check the root-cert.pem in the istio-ca-root-cert config map that Istio propagates for the initial TLS connection. The following example checks the propagated root-cert.pem against the local certificate that you supplied to Vault in the previous section.

  1. Check the Vault version that the management cluster runs.

    kubectl --context="${MGMT_CONTEXT}" exec -n vault vault-0 -- /bin/sh -c "vault version"
    

    Example output:

    Vault v1.11.3 (17250b25303c6418c283c95b1d5a9c9f16174fe8), built 2022-08-26T10:27:10Z
    
  2. Check the root trust policy for errors.

    kubectl describe RootTrustPolicy ${REMOTE_CLUSTER} -n gloo-mesh --context ${REMOTE_CONTEXT}
    
  3. Check the mesh for errors.

    kubectl describe mesh ${MESH} -n gloo-mesh --context ${REMOTE_CONTEXT}
    
  4. From your terminal, navigate to the same directory as the root-cert.pem file that you previously created. Or, if you are using an existing Vault deployment, save the root certificate as root-cert.pem.

  5. Check the difference between the root certificate that istiod uses and the Vault root certificate. If installed correctly, the files are the same.

    kubectl --context=$REMOTE_CONTEXT get cm -n bookinfo istio-ca-root-cert -ojson | jq -r  '.data["root-cert.pem"]' | diff -q root-cert.pem -
    
  6. If you see that the files differ, check the istiod logs.

    kubectl logs -n istio-system --context ${REMOTE_CONTEXT} $(kubectl get pods -n istio-system -l app=istiod --context ${REMOTE_CONTEXT} | cut -d" " -f1 | tail -1) > istiod-logs.txt
    
  7. Check the issued certificates for errors.

    kubectl describe issuedcertificates -n istio-system --context ${REMOTE_CONTEXT}
    

For more troubleshooting steps, see Troubleshoot errors with the Vault setup or Debug Istio.

Rotating certificates for Istio workloads

When certificates are issued, pods that are managed by Istio must be restarted to ensure they pick up the new certificates. The certificate issuer creates a PodBounceDirective, which contains the namespaces and labels of the pods that must be restarted. For more information about how certificate rotation works in Istio, review the video series in this blog post.

Note: To avoid potential downtime for your apps in production, disable the PodBounceDirective feature by setting autoRestartPods to false. Then, control pod restarts in another way, such as a rolling update.

  1. Get your root trust policies.

    kubectl get roottrustpolicy --context ${MGMT_CONTEXT} -A
    
  2. In the root trust policy, remove or set the autoRestartPods field to false.

    kubectl edit roottrustpolicy --context ${MGMT_CONTEXT} -n <namespace> <root-trust-policy>
    
    apiVersion: admin.gloo.solo.io/v2
    kind: RootTrustPolicy
    metadata:
      name: istio-ingressgateway
      namespace: gloo-mesh
    spec:
      config:
        autoRestartPods: false
        ...
    
  3. To ensure pods pick up the new certificates, restart the istiod pod in each remote cluster.

    kubectl --context {$REMOTE_CONTEXT} -n istio-system patch deployment istiod \
        -p "{\"spec\":{\"template\":{\"metadata\":{\"labels\":{\"date\":\"`date +'%s'`\"}}}}}"
    
  4. Restart your app pods that are managed by Istio, such as by using a rolling update strategy.

Troubleshoot errors with the Vault setup

If you have errors with the steps to install Vault, review the following table.

Error Description
Error from server (NotFound): pods “vault-0” not found. Error from server (BadRequest): pod vault-0 does not have a host assigned. The Vault pod might not be running. Check the pod status, troubleshoot any issues, wait for the pod to start, and try again.
* path is already in use You already have set up that path. If you already ran the script, you can ignore this message.
Error writing data to pki/config/ca: Error making API request. Code: 400. Errors: * the given certificate is not marked for CA use and cannot be used with this backend command terminated with exit code 2 If you are using macOS, you might have the default LibreSSL version. Set up OpenSSL instead. For more information see Before you begin.

Example script

You can review or adapt the following example script for your own use.

Environment details:

The script organizes the functions into the following commands that you can run.

  1. Copy the GitHub Gist, also rendered after these steps.

  2. Make sure to update the environment variables at the beginning of the script for the Gloo version, management, and workload cluster contexts that you want to use.

  3. Read and execute the Vault script.

    source ~/Downloads/lib.sh
    
  4. Execute the Vault functions in order. If you notice errors, try running them one at a time, or refer to the troubleshooting section.

    • Run all functions at once:
      vault-install-all
      

    Run functions separately, one at a time:

    1. Install Vault on the management cluster.
      vault-install
      
    2. Enable Vault authentication for Kubernetes.
      vault-enable-kube-auth
      
    3. Set up the CA in Vault.
      vault-setup-ca
      
  5. Verify Vault. Note that this verification assumes you have httpbin on each workload cluster in the httpbin namespace.

    vault-verify
    

Example script gist

GitHub Gist