Learn how the Istio root CA, intermediate CA, and leaf certificates are rotated when you use the Gloo root trust policy, and the options that are available to you to customize this process.
Currently, you cannot automatically rotate the root CA certificate before it expires. However, you can use Gloo Mesh to gradually introduce a second root CA certificate that replaces the old root CA certificate over time. Note that this rotation is a manual process and can only be used if you chose Option 1 or Option 2 of the managed CA setup options. In addition, you can use the same process to replace the first intermediate CA certificate as described in Option 3. If you created a root trust policy in your cluster, and your root certificate is about to expire, contact your Solo account representative to help with rotating the root CA certificate in your cluster.
If you use Gloo self-signed certificates to create the root CA as described in Option 1: Fully managed root and intermediate CAs, the root CA is valid for 1 year by default. You can customize the validity of the root CA by setting the ttlDays in the spec.config.mgmtServerCa.generated section of your root trust policy.
The following example creates a self-signed root CA certificate that is valid for 730 days.
To update your root CA, follow these general steps. Note that rotating the root CA certificate is a multi-day process so that your workloads gradually pick up the new root CA certificate without downtime.
Day 0 represents the initial state of your current setup. In this setup, you have your old root CA certificate that is about to expire and that is currently used to sign intermediate CA certificates for your workload clusters. For setup instructions and a sample root trust policy, see the steps after the following diagram.
You introduce the new root CA certificate as a passive root CA certificate. Passive certificates are used to verify intermediate CA certificates, but cannot be used to sign intermediate CA certificates. Your old root CA certificate is still used to sign intermediate CA certificates. For setup instructions and a sample root trust policy, see the steps after the following diagram.
You introduce your new root CA certificate by using the passiveCertificateAuthorities feature in the Gloo root trust policy. To create a new root CA certificate and key, choose between the following options:
Manually create the root CA certificate and key and store these credentials in a secret. You refer to that secret in the passiveCertificateAuthorities section of your root trust policy. You can use this feature even if you used Gloo self-signed certificates to create your initial root CA certificate as described in Option 1: Fully managed root and intermediate CAs.
Use Gloo Mesh to automatically create a new, self-signed root CA certificate key. Note that this option is not recommended in a production setup.
After introducing the new root CA certificate, restart istiod such as by deleting the pod. During the restart, istiod requests a new intermediate CA certificate. The new intermediate certificate lists both the old and the new root CA certificates as root certificates in the certificate chain. However, the intermediate CA certificate itself was signed by using the old (active) root CA certificate.
To verify that workloads in your mesh can still communicate with each other, restart a non-critical workload or a test workload in your cluster so that it requests a new leaf certificate from istiod. In the certificate chain of the new leaf certificate, the old root CA is listed as the issuer for the intermediate CA certificate.
Send a request from your test workload to a workload in the service mesh that was not yet restarted and verify that these workloads can communicate with each other.
On the next day, you start signing intermediate CA certificates for your workload clusters by using the new root CA certificate. To do that, you must set your new root CA certificate as the active CA certificate and mark your old root CA certificate as the passive one. For setup instructions and a sample root trust policy, see the steps after the following diagram.
In your root trust policy, you swap the old and new root CA certificate. The new root certificate becomes the active root CA certificate that is used to sign intermediate CA certificates. At the same time, the old root certificate is set to passive and can only be used to verify intermediate CA certificates.
After swapping the root CA certificates, restart istiod. During the restart, istiod requests a new intermediate CA certificate. The new intermediate certificate lists both the old and the new root CA certificates as root certificates in the certificate chain. However, the intermediate CA certificate itself is now signed with the new root CA certificate.
To verify that workloads in your mesh can still communicate with each other, restart a non-critical workload or a test workload in your cluster so that it requests a new leaf certificate from istiod. In the certificate chain of the new leaf certificate, the new root CA is listed as the issuer for the intermediate CA certificate.
Send a request from your test workload to a workload in the service mesh that was not yet restarted and verify that these workloads can communicate with each other.
On day 3, you manually restart all the workloads in your service mesh. In the certificate chain, both root CA certificates are listed, but only the new one was used to sign the intermediate CA certificate. Note that because leaf certificates have a default expiration time of 1 day, some of your workloads might already use the new certificate chain. For setup instructions and a sample root trust policy, see the steps after the following diagram.
To make sure all workloads point to the new root CA certificate, restart all the workloads in the service mesh.
Verify that communication between your workloads still works.
Now that all your workloads point to the new root CA in the certificate chain, you can go ahead and remove the old root CA certificate. For setup instructions and a sample root trust policy, see the steps after the following diagram.
In your root trust policy, remove the passiveCertificateAuthorities section in your root trust policy. This step removes your old root CA certificate as a passive certificate that can be used to verify intermediate CA certificates.
Restart istiod to remove the old root CA certificate from the certificate chain.
To verify that workloads in your mesh can still communicate with each other, restart a non-critical workload or a test workload in your cluster so that it requests a new leaf certificate from istiod.
Send a request from your test workload to a workload in the service mesh that was not yet restarted and verify that these workloads can communicate with each other.
Optional: Restart all your workloads in the service mesh to remove the old root CA certificate from the certificate chain. Note that this step is optional as workloads renew their certificates within 24 hours by default.
If not otherwise specified, the intermediate CA certificates that Gloo Mesh generates are valid for 1 year by default. To customize the certificate validity, you can specify the number of days the intermediate certificate is valid for in the ttlDays field of the spec.config.intermediateCertOptions section of the root trust policy. In addition, you specify when you want Gloo Mesh to start the certificate rotation process by setting the spec.config.intermediateCertOptions.secretRotationGracePeriodRatio option.
In the following example root trust policy, the intermediate certificate is valid for one day. Because the grace period ratio is set to 0.1, Gloo Mesh starts rotating the certificate 2.4 hours before the certificate expires (24hx0.1).
The following diagram shows how intermediate CA certificates are rotated behind the scenes when a certificate reconcile event is received in Gloo Mesh.
If you installed Istio with the AUTO_RELOAD_PLUGIN_CERTS=true option, the Istio control plane istiod automatically picks up new intermediate CA certificates and uses these certificates to sign the leaf certificates for the workloads. No restart of the istiod pod is required.
However, if you did not or do not want to enable this option, you must manually restart the istiod pod to apply the intermediate CA changes.
By default, leaf certificates that are issued for the workloads in the service mesh are valid for 24 hours. You can change this value by setting the proxyMetadata.SECRET_TTL value in your Istio operator configuration, such as by upgrading Gloo Mesh-managed Istio. For example, to issue certificates that are valid for 12 hours, set the proxyMetadata.SECRET_TTL value to 12h0m0s.