To run an Istio service mesh across clusters, you must establish trust between workloads across cluster boundaries. By default, Istio sets up the Istio control plane istiod as the root certificate authority (CA) so that issues leaf certificates can be issued to the workloads in a service mesh. However, this setup gets very complex in a multicluster service mesh as all certificates must be derived from the same root.

Use this page to learn about the default Istio certificate setup, its challenges, and how you can use the Gloo Mesh Enterprise root trust policy to federate trust in a multicluster service mesh.

Default Istio root CA setup

When you install Istio without Gloo Mesh Core, the Istio control plane istiod automatically creates a private key and self-signed root certificate that is valid for 10 years and stored in the cacerts Kubernetes secret on the cluster. Istiod is then configured with this secret and set up as the root certificate authority (CA) for the service mesh.

As the root CA, istiod uses the private root CA key to sign and issue leaf certificates to the workloads in the service mesh. The leaf certificates encode a workload’s identity and can therefore be used to enforce authentication and secure communication between workloads via mutual TLS (mTLS) as shown in the following image.

Figure: Default Istio root CA setup


Because all of the leaf certificates share the same root, trust can be established between the workloads in the service mesh. Similarly, if the root CA certificate is rotated, and the change is not reflected in all of the leaf certificates, communication between workloads starts to break until all leaf certificates are updated with the new root CA certificate.

Challenges with the default Istio setup

The default Istio root CA approach is good when you begin to get started with Istio. However, this setup can get very complex and hard to manage, especially when your service mesh spreads across multiple clusters. To allow communication between workloads on different clusters, you must ensure that all workloads share the same root certificate and certificate chain so that trust can be established across clusters. Instead of treating istiod as the root CA, you can use your preferred private key infrastructure (PKI) provider to derive intermediate certificates from the root certificate. The intermediate certificate and key are then stored in the cacert Kubernetes secret and configure istiod as an intermediate CA as shown in the following image.

Figure: Default Istio root CA setup


While this approach allows you to protect and manage the root CA key in your preferred PKI provider, you must manually provision intermediate CA certificates and keys for all your clusters, and keep track of when these certificates expire so that you can manually rotate them.

Managed Istio CA lifecycle with Gloo Mesh Enterprise’s root trust policy

With the root trust policy in Gloo Mesh Enterprise, you can automatically set up an Istio root CA and use that root CA to automatically issue intermediate CA certificates for all your workload clusters that make up your multicluster service mesh. Because all intermediate CA certificates are derived from the same root, your workloads can securely talk to each other, even across cluster boundaries. This approach is also referred to as federated trust.

Gloo Mesh Enterprise provides the option to fully or partially manage the Istio CA certificate lifecycle for you. The option that is right for you depends on the type of environment that you have and the level of control you want to have over the Istio CA certificate lifecycle.

To learn more about your managed CA options, see Managed CAs in the Gloo Mesh Enterprise docs.

Bring your own Istio CAs

To enhance the security of your setup even further and have full control over the Istio CA lifecycle, you can generate and store the root and intermediate CA certificates and keys with your own PKI provider. You can then use tools, such as cert-manager to send certificate signing requests on behalf of istiod to your PKI provider. Cert-manager stores the signed intermediate certificates and keys in the cacerts Kubernetes secret so that istiod can use these credentials to issue leaf certificates for the workloads in the service mesh. You can set up cert-manager to also check the certificates and renew them before they expire.

Refer to the AWS and cert-manager guide to learn how to use cert-manager with an external PKI provider.