Root certificate rotation

Expiration settings

If you use Gloo self-signed certificates to create the root CA as described in Option 1: Fully managed root and intermediate CAs, the root CA is valid for 1 year by default. You can customize the validity of the root CA by setting the ttlDays in the spec.config.mgmtServerCa.generated section of your root trust policy.

The following example creates a self-signed root CA certificate that is valid for 730 days.

  apiVersion: admin.gloo.solo.io/v2
kind: RootTrustPolicy
metadata:
  name: root-trust-policy
  namespace: gloo-mesh
spec:
  config:
    mgmtServerCa: 
      generated:
        ttlDays: 730

Certificate rotation process

To update your root CA, follow these general steps. Note that rotating the root CA certificate is a multi-day process so that your workloads gradually pick up the new root CA certificate without downtime.

Day 0 represents the initial state of your current setup. In this setup, you have your old root CA certificate that is about to expire and that is currently used to sign intermediate CA certificates for your workload clusters. For setup instructions and a sample root trust policy, see the steps after the following diagram.

Figure: Day 0: Initial setup
Figure: Day 0: Initial setup
Figure: Day 0: Initial setup
Figure: Day 0: Initial setup

Sample root trust policy:

  apiVersion: admin.gloo.solo.io/v2
kind: RootTrustPolicy
metadata:
  name: root-trust-policy
  namespace: gloo-mesh
spec:
  config:
    mgmtServerCa: 
      secretRef:
        name: root-certificate-old
        namespace: gloo-mesh

You introduce the new root CA certificate as a passive root CA certificate. Passive certificates are used to verify intermediate CA certificates, but cannot be used to sign intermediate CA certificates. Your old root CA certificate is still used to sign intermediate CA certificates. For setup instructions and a sample root trust policy, see the steps after the following diagram.

Figure: Day 1: Introduce a new root CA
Figure: Day 1: Introduce a new root CA
Figure: Day 1: Introduce a new root CA
Figure: Day 1: Introduce a new root CA
  1. You introduce your new root CA certificate by using the passiveCertificateAuthorities feature in the Gloo root trust policy. To create a new root CA certificate and key, choose between the following options:
    • Manually create the root CA certificate and key and store these credentials in a secret. You refer to that secret in the passiveCertificateAuthorities section of your root trust policy. You can use this feature even if you used Gloo self-signed certificates to create your initial root CA certificate as described in Option 1: Fully managed root and intermediate CAs.
    • Use Gloo Mesh to automatically create a new, self-signed root CA certificate key. Note that this option is not recommended in a production setup.
  2. After introducing the new root CA certificate, restart istiod such as by deleting the pod. During the restart, istiod requests a new intermediate CA certificate. The new intermediate certificate lists both the old and the new root CA certificates as root certificates in the certificate chain. However, the intermediate CA certificate itself was signed by using the old (active) root CA certificate.
  3. To verify that workloads in your mesh can still communicate with each other, restart a non-critical workload or a test workload in your cluster so that it requests a new leaf certificate from istiod. In the certificate chain of the new leaf certificate, the old root CA is listed as the issuer for the intermediate CA certificate.
  4. Send a request from your test workload to a workload in the service mesh that was not yet restarted and verify that these workloads can communicate with each other.

Sample root trust policy:

  apiVersion: admin.gloo.solo.io/v2
kind: RootTrustPolicy
metadata:
  name: istio-ingressgateway
  namespace: gloo-mesh
spec:
  config:
    mgmtServerCa:
      secretRef:
        name: root-certificate-old
        namespace: gloo-mesh
    passiveCertificateAuthorities:
      - secretRef:
          name: root-certificate-new
          namespace: gloo-mesh

On the next day, you start signing intermediate CA certificates for your workload clusters by using the new root CA certificate. To do that, you must set your new root CA certificate as the active CA certificate and mark your old root CA certificate as the passive one. For setup instructions and a sample root trust policy, see the steps after the following diagram.

Figure: Day 2: Begin signing intermediate CA certificates with the new root CA
Figure: Day 2: Begin signing intermediate CA certificates with the new root CA
Figure: Day 2: Begin signing intermediate CA certificates with the new root CA
Figure: Day 2: Begin signing intermediate CA certificates with the new root CA
  1. In your root trust policy, you swap the old and new root CA certificate. The new root certificate becomes the active root CA certificate that is used to sign intermediate CA certificates. At the same time, the old root certificate is set to passive and can only be used to verify intermediate CA certificates.
  2. After swapping the root CA certificates, restart istiod. During the restart, istiod requests a new intermediate CA certificate. The new intermediate certificate lists both the old and the new root CA certificates as root certificates in the certificate chain. However, the intermediate CA certificate itself is now signed with the new root CA certificate.
  3. To verify that workloads in your mesh can still communicate with each other, restart a non-critical workload or a test workload in your cluster so that it requests a new leaf certificate from istiod. In the certificate chain of the new leaf certificate, the new root CA is listed as the issuer for the intermediate CA certificate.
  4. Send a request from your test workload to a workload in the service mesh that was not yet restarted and verify that these workloads can communicate with each other.

Sample root trust policy:

  apiVersion: admin.gloo.solo.io/v2
kind: RootTrustPolicy
metadata:
  name: istio-ingressgateway
  namespace: gloo-mesh
spec:
  config:
    mgmtServerCa:
      secretRef:
        name: root-certificate-new
        namespace: gloo-mesh
    passiveCertificateAuthorities:
      - secretRef:
          name: root-certificate-old
          namespace: gloo-mesh

On day 3, you manually restart all the workloads in your service mesh. In the certificate chain, both root CA certificates are listed, but only the new one was used to sign the intermediate CA certificate. Note that because leaf certificates have a default expiration time of 1 day, some of your workloads might already use the new certificate chain. For setup instructions and a sample root trust policy, see the steps after the following diagram.

Figure: Day 3: Update all workloads to point to the new root CA
Figure: Day 3: Update all workloads to point to the new root CA
Figure: Day 3: Update all workloads to point to the new root CA
Figure: Day 3: Update all workloads to point to the new root CA
  1. To make sure all workloads point to the new root CA certificate, restart all the workloads in the service mesh.
  2. Verify that communication between your workloads still works.

Sample root trust policy:

  apiVersion: admin.gloo.solo.io/v2
kind: RootTrustPolicy
metadata:
  name: istio-ingressgateway
  namespace: gloo-mesh
spec:
  config:
    mgmtServerCa:
      secretRef:
        name: root-certificate-new
        namespace: gloo-mesh
    passiveCertificateAuthorities:
      - secretRef:
          name: root-certificate-old
          namespace: gloo-mesh

Now that all your workloads point to the new root CA in the certificate chain, you can go ahead and remove the old root CA certificate. For setup instructions and a sample root trust policy, see the steps after the following diagram.

Figure: Day 4: Remove the old root CA
Figure: Day 4: Remove the old root CA
Figure: Day 4: Remove the old root CA
Figure: Day 4: Remove the old root CA
  1. In your root trust policy, remove the passiveCertificateAuthorities section in your root trust policy. This step removes your old root CA certificate as a passive certificate that can be used to verify intermediate CA certificates.
  2. Restart istiod to remove the old root CA certificate from the certificate chain.
  3. To verify that workloads in your mesh can still communicate with each other, restart a non-critical workload or a test workload in your cluster so that it requests a new leaf certificate from istiod.
  4. Send a request from your test workload to a workload in the service mesh that was not yet restarted and verify that these workloads can communicate with each other.
  5. Optional: Restart all your workloads in the service mesh to remove the old root CA certificate from the certificate chain. Note that this step is optional as workloads renew their certificates within 24 hours by default.

Sample root trust policy:

  apiVersion: admin.gloo.solo.io/v2
kind: RootTrustPolicy
metadata:
  name: istio-ingressgateway
  namespace: gloo-mesh
spec:
  config:
    mgmtServerCa:
      secretRef:
        name: root-certificate-new
        namespace: gloo-mesh

Intermediate certificate rotation

Automatic intermediate CA rotation is enabled for all managed CA setups that are described in the Setup options.

Rotation and expiration settings

If not otherwise specified, the intermediate CA certificates that Gloo Mesh generates are valid for 1 year by default. To customize the certificate validity, you can specify the number of days the intermediate certificate is valid for in the ttlDays field of the spec.config.intermediateCertOptions section of the root trust policy. In addition, you specify when you want Gloo Mesh to start the certificate rotation process by setting the spec.config.intermediateCertOptions.secretRotationGracePeriodRatio option.

In the following example root trust policy, the intermediate certificate is valid for one day. Because the grace period ratio is set to 0.1, Gloo Mesh starts rotating the certificate 2.4 hours before the certificate expires (24hx0.1).

  apiVersion: admin.gloo.solo.io/v2
kind: RootTrustPolicy
metadata:
  name: root-trust-policy
  namespace: gloo-mesh
spec:
  config:
    intermediateCertOptions:
      secretRotationGracePeriodRatio: 0.1
      ttlDays: 1
    mgmtServerCa: 
      generated:
        ttlDays: 730

Certificate rotation process

The following diagram shows how intermediate CA certificates are rotated behind the scenes when a certificate reconcile event is received in Gloo Mesh.

Figure: Automatic intermediate CA certificate rotation process in Gloo Mesh Enterprise
Figure: Automatic intermediate CA certificate rotation process in Gloo Mesh Enterprise
Figure: Automatic intermediate CA certificate rotation process in Gloo Mesh Enterprise
Figure: Automatic intermediate CA certificate rotation process in Gloo Mesh Enterprise

Rollout of updated certificates

If you installed Istio with the AUTO_RELOAD_PLUGIN_CERTS=true option, the Istio control plane istiod automatically picks up new intermediate CA certificates and uses these certificates to sign the leaf certificates for the workloads. No restart of the istiod pod is required.

However, if you did not or do not want to enable this option, you must manually restart the istiod pod to apply the intermediate CA changes.

Leaf certificate rotation

By default, leaf certificates that are issued for the workloads in the service mesh are valid for 24 hours. You can change this value by setting the proxyMetadata.SECRET_TTL value in your Istio operator configuration, such as by upgrading Gloo Mesh-managed Istio. For example, to issue certificates that are valid for 12 hours, set the proxyMetadata.SECRET_TTL value to 12h0m0s.