Certificate rotation overview
Learn how the Istio root CA, intermediate CA, and leaf certificates are rotated when you use the Gloo Mesh root trust policy, and the options that are available to you to customize this process.
Root certificate rotation
Currently, you cannot automatically rotate the root CA certificate before it expires. However, you can use Gloo Mesh to gradually introduce a second root CA certificate that replaces the old root CA certificate over time. Note that this rotation is a manual process and can only be used if you chose Option 1 or Option 2 of the managed CA setup options. In addition, you can use the same process to replace the first intermediate CA certificate as described in Option 3. If you created a root trust policy in your cluster, and your root certificate is about to expire, contact your Solo account representative to help with rotating the root CA certificate in your cluster.
Expiration settings
If you use Gloo Mesh self-signed certificates to create the root CA as described in Option 1: Fully managed root and intermediate CAs, the root CA is valid for 1 year by default. You can customize the validity of the root CA by setting the ttlDays
in the spec.config.mgmtServerCa.generated
section of your root trust policy.
The following example creates a self-signed root CA certificate that is valid for 730 days.
apiVersion: admin.gloo.solo.io/v2
kind: RootTrustPolicy
metadata:
name: root-trust-policy
namespace: gloo-mesh
spec:
config:
mgmtServerCa:
generated:
ttlDays: 730
Certificate rotation process
To update your root CA, follow these general steps. Note that rotating the root CA certificate is a multi-day process so that your workloads gradually pick up the new root CA certificate without downtime.
- Day 0 - Initial state
- Day 1 - Introduce new root CA
- Day 2 - Begin signing with the new root CA
- Day 3 - Update all workloads
- Day 4 - Remove the old root CA
Day 0 represents the initial state of your current setup. In this setup, you have your old root CA certificate that is about to expire and that is currently used to sign intermediate CA certificates for your workload clusters. For setup instructions and a sample root trust policy, see the steps after the following diagram.
Sample root trust policy:
apiVersion: admin.gloo.solo.io/v2
kind: RootTrustPolicy
metadata:
name: root-trust-policy
namespace: gloo-mesh
spec:
config:
mgmtServerCa:
secretRef:
name: root-certificate-old
namespace: gloo-mesh
You introduce the new root CA certificate as a passive root CA certificate. Passive certificates are used to verify intermediate CA certificates, but cannot be used to sign intermediate CA certificates. Your old root CA certificate is still used to sign intermediate CA certificates. For setup instructions and a sample root trust policy, see the steps after the following diagram.
- You introduce your new root CA certificate by using the
passiveCertificateAuthorities
feature in the Gloo Mesh root trust policy. To create a new root CA certificate and key, choose between the following options:- Manually create the root CA certificate and key and store these credentials in a secret. You refer to that secret in the
passiveCertificateAuthorities
section of your root trust policy. You can use this feature even if you used Gloo Mesh self-signed certificates to create your initial root CA certificate as described in Option 1: Fully managed root and intermediate CAs. - Use Gloo Mesh to automatically create a new, self-signed root CA certificate key. Note that this option is not recommended in a production setup.
- Manually create the root CA certificate and key and store these credentials in a secret. You refer to that secret in the
- After introducing the new root CA certificate, restart istiod such as by deleting the pod. During the restart, istiod requests a new intermediate CA certificate. The new intermediate certificate lists both the old and the new root CA certificates as root certificates in the certificate chain. However, the intermediate CA certificate itself was signed by using the old (active) root CA certificate.
- To verify that workloads in your mesh can still communicate with each other, restart a non-critical workload or a test workload in your cluster so that it requests a new leaf certificate from istiod. In the certificate chain of the new leaf certificate, the old root CA is listed as the issuer for the intermediate CA certificate.
- Send a request from your test workload to a workload in the service mesh that was not yet restarted and verify that these workloads can communicate with each other.
Sample root trust policy:
apiVersion: admin.gloo.solo.io/v2
kind: RootTrustPolicy
metadata:
name: istio-ingressgateway
namespace: gloo-mesh
spec:
config:
mgmtServerCa:
secretRef:
name: root-certificate-old
namespace: gloo-mesh
passiveCertificateAuthorities:
- secretRef:
name: root-certificate-new
namespace: gloo-mesh
On the next day, you start signing intermediate CA certificates for your workload clusters by using the new root CA certificate. To do that, you must set your new root CA certificate as the active CA certificate and mark your old root CA certificate as the passive one. For setup instructions and a sample root trust policy, see the steps after the following diagram.
- In your root trust policy, you swap the old and new root CA certificate. The new root certificate becomes the active root CA certificate that is used to sign intermediate CA certificates. At the same time, the old root certificate is set to passive and can only be used to verify intermediate CA certificates.
- After swapping the root CA certificates, restart istiod. During the restart, istiod requests a new intermediate CA certificate. The new intermediate certificate lists both the old and the new root CA certificates as root certificates in the certificate chain. However, the intermediate CA certificate itself is now signed with the new root CA certificate.
- To verify that workloads in your mesh can still communicate with each other, restart a non-critical workload or a test workload in your cluster so that it requests a new leaf certificate from istiod. In the certificate chain of the new leaf certificate, the new root CA is listed as the issuer for the intermediate CA certificate.
- Send a request from your test workload to a workload in the service mesh that was not yet restarted and verify that these workloads can communicate with each other.
Sample root trust policy:
apiVersion: admin.gloo.solo.io/v2
kind: RootTrustPolicy
metadata:
name: istio-ingressgateway
namespace: gloo-mesh
spec:
config:
mgmtServerCa:
secretRef:
name: root-certificate-new
namespace: gloo-mesh
passiveCertificateAuthorities:
- secretRef:
name: root-certificate-old
namespace: gloo-mesh
On day 3, you manually restart all the workloads in your service mesh. In the certificate chain, both root CA certificates are listed, but only the new one was used to sign the intermediate CA certificate. Note that because leaf certificates have a default expiration time of 1 day, some of your workloads might already use the new certificate chain. For setup instructions and a sample root trust policy, see the steps after the following diagram.
- To make sure all workloads point to the new root CA certificate, restart all the workloads in the service mesh.
- Verify that communication between your workloads still works.
Sample root trust policy:
apiVersion: admin.gloo.solo.io/v2
kind: RootTrustPolicy
metadata:
name: istio-ingressgateway
namespace: gloo-mesh
spec:
config:
mgmtServerCa:
secretRef:
name: root-certificate-new
namespace: gloo-mesh
passiveCertificateAuthorities:
- secretRef:
name: root-certificate-old
namespace: gloo-mesh
Now that all your workloads point to the new root CA in the certificate chain, you can go ahead and remove the old root CA certificate. For setup instructions and a sample root trust policy, see the steps after the following diagram.
- In your root trust policy, remove the
passiveCertificateAuthorities
section in your root trust policy. This step removes your old root CA certificate as a passive certificate that can be used to verify intermediate CA certificates. - Restart istiod to remove the old root CA certificate from the certificate chain.
- To verify that workloads in your mesh can still communicate with each other, restart a non-critical workload or a test workload in your cluster so that it requests a new leaf certificate from istiod.
- Send a request from your test workload to a workload in the service mesh that was not yet restarted and verify that these workloads can communicate with each other.
- Optional: Restart all your workloads in the service mesh to remove the old root CA certificate from the certificate chain. Note that this step is optional as workloads renew their certificates within 24 hours by default.
Sample root trust policy:
apiVersion: admin.gloo.solo.io/v2
kind: RootTrustPolicy
metadata:
name: istio-ingressgateway
namespace: gloo-mesh
spec:
config:
mgmtServerCa:
secretRef:
name: root-certificate-new
namespace: gloo-mesh
Intermediate certificate rotation
Automatic intermediate CA rotation is enabled for all managed CA setups that are described in the Setup options.
Rotation and expiration settings
If not otherwise specified, the intermediate CA certificates that Gloo Mesh generates are valid for 1 year by default. To customize the certificate validity, you can specify the number of days the intermediate certificate is valid for in the ttlDays
field of the spec.config.intermediateCertOptions
section of the root trust policy. In addition, you specify when you want Gloo Mesh to start the certificate rotation process by setting the spec.config.intermediateCertOptions.secretRotationGracePeriodRatio
option.
In the following example root trust policy, the intermediate certificate is valid for one day. Because the grace period ratio is set to 0.1, Gloo Mesh starts rotating the certificate 2.4 hours before the certificate expires (24hx0.1).
apiVersion: admin.gloo.solo.io/v2
kind: RootTrustPolicy
metadata:
name: root-trust-policy
namespace: gloo-mesh
spec:
config:
autoRestartPods: true
intermediateCertOptions:
secretRotationGracePeriodRatio: 0.1
ttlDays: 1
mgmtServerCa:
generated:
ttlDays: 730
Certificate rotation process
The following diagram shows how intermediate CA certificates are rotated behind the scenes when a certificate reconcile event is received in Gloo Mesh.
Rollout of updated certificates
If you installed Istio with the AUTO_RELOAD_PLUGIN_CERTS=true
option, the Istio control plane istiod automatically picks up new intermediate CA certificates and uses these certificates to sign the leaf certificates for the workloads. No restart of the istiod pod is required.
However, if you did not or do not want to enable this option, you must manually restart the istiod pod to apply the intermediate CA changes.
Leaf certificate rotation
By default, leaf certificates that are issued for the workloads in the service mesh are valid for 24 hours. You can change this value by setting the proxyMetadata.SECRET_TTL
value in your Istio operator configuration, such as by upgrading Gloo Mesh-managed Istio. For example, to issue certificates that are valid for 12 hours, set the proxyMetadata.SECRET_TTL
value to 12h0m0s.