Gloo Mesh provides the option to fully or partially manage the Istio CA certificate lifecycle for you as shown in the following table. The option that is right for you depends on the type of environment that you have and the level of control you want to have over the Istio certificate lifecycle.

#OptionDescriptionRoot CA in cluster?Certificate rotationSuggested environment
1Fully managed root and intermediate CAsUse Gloo Mesh to generate a self-signed root CA certificate and store it on the Gloo management cluster. When a new workload cluster is registered with the management cluster, the root CA is used to sign and issue an intermediate CA certificate that is stored on the workload cluster.YesAutomatic intermediate and leaf certificate rotationTesting or POCs
2Managed intermediate CAInstead of using Gloo Mesh self-signed certificates, you can create your own root CA certificate and private key by using your preferred PKI provider. Then, you store the private key and certificate in a Kubernetes secret on the Gloo management cluster to enable Gloo Mesh to automatically sign and issue intermediate CA certificates for your workload clusters based on the same root.YesAutomatic intermediate and leaf certificate rotationTesting or POCs
3Managed multi-intermediate CAsThis setup is similar to the managed intermediate CA option. Instead of setting up a root CA in the Gloo management cluster, you add an additional intermediate CA layer to your certificate chain. The root CA key and certificate are generated and stored with your preferred PKI provider. You then use the root CA private key to sign and issue an intermediate CA certificate and key. The intermediate CA certificate and key are then stored in the Gloo management cluster so that Gloo Mesh can automatically issue intermediate CA certificates and keys for each workload cluster that is registered.NoOnly intermediate CA certificates in the workload clusters can be automatically rotated. Automatic rotation of the intermediate CA certificate in the management cluster or the root CA in the external PKI provider is not supported.Production
4Integrate with VaultIn this setup, you store and manage the root CA certificate and key in Vault. To enable Gloo Mesh to automatically derive intermediate CA certificates from the root CA in Vault, istiod is injected with the istiod-agent sidecar that authenticates and sends certificate signing requests to Vault. Instead of writing the intermediate CA certificate and key to the cacerts Kubernetes secret, the istiod-agent stores the credentials in memory, which adds an additional security layer to this approach.Depends on your setupAutomatic intermediate CA and leaf certificate rotation.Production

Option 1: Fully managed root and intermediate CAs

Create a root trust policy in Gloo Mesh Enterprise to automatically set up a managed Istio root certificate authority (CA) with Gloo Mesh self-signed certificates in the management cluster as shown in the following image. This root CA is then used to automatically sign and issue intermediate CA certificates for each workload cluster.

Figure: Root trust policy components and process when using Gloo Mesh self-signed root CA certificates
Figure: Root trust policy components and process when using Gloo Mesh self-signed root CA certificates
  1. When you create a root trust policy, Gloo Mesh generates and self-signs a root certificate. This root certificate is stored in the root-trust-policy.gloo-mesh Kubernetes secret in the gloo-mesh namespace on the management cluster, and represents the Istio root certificate authority (CA).
  2. When a workload cluster is registered with the management cluster, the Gloo agent generates a private key and sends a certificate signing request (CSR) to the certificate issuer service in the management cluster.
  3. The certificate issuer service verifies the request and uses the root CA key to sign and issue an intermediate certificate and key.
  4. The intermediate certificate and key is returned to the Gloo agent.
  5. The Gloo agent stores the intermediate certificate and key in the cacerts Kubernetes secret in the istio-system namespace on the workload cluster. Then, the Istio control plane istiod is set up to use this secret for future certificate signing requests from the workloads in the service mesh. Istiod therefore acts as an intermediate CA.
  6. Workloads in the service mesh send a CSR to istiod.
  7. Istiod uses the intermediate CA’s private key to sign and issue a leaf certificate.
  8. The leaf certificate is returned to the workload and represents a workload’s identity. When you inspect a workload’s leaf certificate, you can see the entire certificate chain, including the root, intermediate, and leaf certificates.
  9. The sidecar proxies of each workload use the leaf certificate to securely authenticate and communicate with other workloads in the service mesh via mTLS.

To set up this option, see Manage the entire Istio CA lifecycle.

Option 2: Managed intermediate CA

Instead of using Gloo Mesh self-signed certificates for the root CA certificate, you can generate your own Istio root CA certificate and key with the certificate management tool of your choice, and store the certificate and key in a Kubernetes secret on the management cluster. You can then configure Gloo Mesh to use this secret when signing and issuing intermediate certificates to the workload clusters as shown in the following diagram.

Figure: Root trust policy components and process when bringing your own root CA certificate and key
Figure: Root trust policy components and process when bringing your own root CA certificate and key
  1. You generate a root CA certificate and private key by using the certificate management tool of your choice.
  2. You then store the root CA certificate and key in a Kubernetes secret on the Gloo management cluster and reference this secret in the root trust policy.
  3. When a workload cluster is registered with the management cluster, the Gloo agent generates a private key and sends a certificate signing request (CSR) to the certificate issuer service in the management cluster.
  4. The certificate issuer service verifies the request and uses the root CA key that you provided in the Kubernetes secret to sign and issue an intermediate certificate.
  5. The intermediate certificate is returned to the Gloo agent.
  6. The Gloo agent stores the intermediate certificate and key in the cacert Kubernetes secret in the istio-system namespace on the workload cluster. Then, the Istio control plane istiod is set up to use this secret for future certificate signing requests from the workloads in the service mesh. Istiod therefore acts as an intermediate CA.
  7. Workloads in the service mesh send a CSR to istiod.
  8. Istiod uses the intermediate CA’s private key to sign and issue a leaf certificate.
  9. The leaf certificate is returned to the workload and represents a workload’s identity. When you inspect a workload’s leaf certificate, you can see the entire certificate chain, including the root, intermediate, and leaf certificates.
  10. The sidecar proxies of each workload use the leaf certificate to securely authenticate and communicate with other workloads in the service mesh via mTLS.

To set up this option, see Manage Istio intermediate CAs.

Option 3: Managed multi-intermediate CAs

This setup is similar to the managed intermediate CA option. Instead of setting up a root CA in the Gloo management cluster, you add an additional intermediate CA layer to your certificate chain. The root CA key and certificate are generated and stored with your preferred PKI provider. You then use the root CA private key to sign and issue an intermediate CA certificate and key. The intermediate CA certificate and key are then stored in the Gloo management cluster so that Gloo Mesh can automatically issue intermediate CA certificates and keys for each workload cluster that is registered.

Figure: Root trust policy components and process when using multiple intermediate CA layers
Figure: Root trust policy components and process when using multiple intermediate CA layers
  1. The root CA certificate and key are generated and stored with your preferred PKI provider.
  2. The root CA key is used to generate and sign an intermediate CA certificate and key. The certificate and key are stored in a Kubernetes secret on the Gloo management cluster. This secret is then referenced in the Gloo Mesh root trust policy.
  3. When a workload cluster is registered with the management cluster, the Gloo agent generates a private key and sends a certificate signing request (CSR) to the certificate issuer service in the management cluster.
  4. The certificate issuer service verifies the request and uses the intermediate CA key that you provided in the Kubernetes secret to sign and issue a second intermediate certificate.
  5. The intermediate certificate is returned to the Gloo agent.
  6. The Gloo agent stores the intermediate certificate and key in the cacerts Kubernetes secret in the istio-system namespace on the workload cluster. Then, the Istio control plane istiod is set up to use this secret for future certificate signing requests from the workloads in the service mesh. Istiod therefore acts as an intermediate CA.
  7. Workloads in the service mesh send a CSR to istiod.
  8. Istiod uses the intermediate CA’s private key to sign and issue a leaf certificate.
  9. The leaf certificate is returned to the workload and represents a workload’s identity. When you inspect a workload’s leaf certificate, you can see the entire certificate chain, including the root, intermediate, and leaf certificates.
  10. The sidecar proxies of each workload use the leaf certificate to securely authenticate and communicate with other workloads in the service mesh via mTLS.

To set up this option, see Manage Istio intermediate CAs.

Option 4: Integrate with Vault

In this option, you set up a Vault instance in the Gloo management cluster to generate, store, and manage the root CA certificate and key. To automatically derive and issue intermediate CA certificates for the workload clusters, Gloo Mesh configures the istiod-agent, which runs as a sidecar in the istiod pod, and communicates with the Vault instance to request private keys and to sign intermediate certificates for the workload clusters. The private key is directly loaded into the pod filesystem, thereby allowing for an added layer of security by not saving the key to etcd or any permanent storage. When the pod is deleted, the private key is also deleted.

Figure: Root trust policy components and process when storing the root CA certificate and key in Vault
Figure: Root trust policy components and process when storing the root CA certificate and key in Vault
  1. You generate and store your Istio root CA certificate and private key with Vault. Vault can either be set up in your Gloo management cluster or you can use a Vault instance that exists outside your Gloo environment.
  2. In your root trust policy you instruct the Gloo agent to set up the istiod-agent sidecar and configure it with the credential for how to authenticate with Vault.
  3. When a workload cluster is registered with the management cluster, the istiod-agent generates a private key and sends a certificate signing request (CSR) to the Vault instance in the management cluster.
  4. Vault verifies the request and uses the private root CA key to sign and issue an intermediate CA certificate. The intermediate CA certificate is returned to the istiod-agent sidecar. Instead of persisting the intermediate CA certificate and private key in the cacerts Kubernetes secret, the istiod-agent sidecar stores these credentials in memory, which adds an additional security layer to this approach. The private key is lost and needs to be re-created when the istiod-agent pod crashes or restarts.
  5. Workloads in the service mesh send a CSR to istiod.
  6. Istiod verifies the request. Instead of obtaining the intermediate CA private key from the cacerts Kubernetes secret for signing, istiod retrieves the private key from the istiod-agent sidecar directly. Istiod uses the intermediate CA key to sign and issue the leaf certificate.
  7. The leaf certificate is returned to the workload and represents a workload’s identity. When you inspect a workload’s leaf certificate, you can see the entire certificate chain, including the root, intermediate, and leaf certificates.
  8. The sidecar proxies of each workload use the leaf certificate to securely authenticate and communicate with other workloads in the service mesh via mTLS.

To set up this option, see Integrate with Vault.