Certificate rotation overview

You introduce the new root CA certificate as a passive root CA certificate. Passive certificates are used to verify intermediate CA certificates, but cannot be used to sign intermediate CA certificates. Your old root CA certificate is still used to sign intermediate CA certificates. For setup instructions and a sample root trust policy, see the steps after the following diagram.

1. You introduce your new root CA certificate by using the passiveCertificateAuthorities feature in the Gloo root trust policy. To create a new root CA certificate and key, choose between the following options:
   • Manually create the root CA certificate and key and store these credentials in a secret. You refer to that secret in the passiveCertificateAuthorities section of your root trust policy. You can use this feature even if you used Gloo self-signed certificates to create your initial root CA certificate as described in Option 1: Fully managed root and intermediate CAs.
   • Use Gloo Mesh to automatically create a new, self-signed root CA certificate key. Note that this option is not recommended in a production setup.
2. After introducing the new root CA certificate, restart istiod such as by deleting the pod. During the restart, istiod requests a new intermediate CA certificate. The new intermediate certificate lists both the old and the new root CA certificates as root certificates in the certificate chain. However, the intermediate CA certificate itself was signed by using the old (active) root CA certificate.
3. To verify that workloads in your mesh can still communicate with each other, restart a non-critical workload or a test workload in your cluster so that it requests a new leaf certificate from istiod. In the certificate chain of the new leaf certificate, the old root CA is listed as the issuer for the intermediate CA certificate.
4. Send a request from your test workload to a workload in the service mesh that was not yet restarted and verify that these workloads can communicate with each other.

Sample root trust policy:

  apiVersion: admin.gloo.solo.io/v2
kind: RootTrustPolicy
metadata:
  name: istio-ingressgateway
  namespace: gloo-mesh
spec:
  config:
    mgmtServerCa:
      secretRef:
        name: root-certificate-old
        namespace: gloo-mesh
    passiveCertificateAuthorities:
      - secretRef:
          name: root-certificate-new
          namespace: gloo-mesh
  

On the next day, you start signing intermediate CA certificates for your workload clusters by using the new root CA certificate. To do that, you must set your new root CA certificate as the active CA certificate and mark your old root CA certificate as the passive one. For setup instructions and a sample root trust policy, see the steps after the following diagram.

1. In your root trust policy, you swap the old and new root CA certificate. The new root certificate becomes the active root CA certificate that is used to sign intermediate CA certificates. At the same time, the old root certificate is set to passive and can only be used to verify intermediate CA certificates.
2. After swapping the root CA certificates, restart istiod. During the restart, istiod requests a new intermediate CA certificate. The new intermediate certificate lists both the old and the new root CA certificates as root certificates in the certificate chain. However, the intermediate CA certificate itself is now signed with the new root CA certificate.
3. To verify that workloads in your mesh can still communicate with each other, restart a non-critical workload or a test workload in your cluster so that it requests a new leaf certificate from istiod. In the certificate chain of the new leaf certificate, the new root CA is listed as the issuer for the intermediate CA certificate.
4. Send a request from your test workload to a workload in the service mesh that was not yet restarted and verify that these workloads can communicate with each other.

Sample root trust policy:

  apiVersion: admin.gloo.solo.io/v2
kind: RootTrustPolicy
metadata:
  name: istio-ingressgateway
  namespace: gloo-mesh
spec:
  config:
    mgmtServerCa:
      secretRef:
        name: root-certificate-new
        namespace: gloo-mesh
    passiveCertificateAuthorities:
      - secretRef:
          name: root-certificate-old
          namespace: gloo-mesh
  

On day 3, you manually restart all the workloads in your service mesh. In the certificate chain, both root CA certificates are listed, but only the new one was used to sign the intermediate CA certificate. Note that because leaf certificates have a default expiration time of 1 day, some of your workloads might already use the new certificate chain. For setup instructions and a sample root trust policy, see the steps after the following diagram.

1. To make sure all workloads point to the new root CA certificate, restart all the workloads in the service mesh.
2. Verify that communication between your workloads still works.

Sample root trust policy:

  \{\{< prism lang="yaml" line="(.*)" >\}\}
apiVersion: admin.gloo.solo.io/v2
kind: RootTrustPolicy
metadata:
  name: istio-ingressgateway
  namespace: gloo-mesh
spec:
  config:
    mgmtServerCa:
      secretRef:
        name: root-certificate-new
        namespace: gloo-mesh
    passiveCertificateAuthorities:
      - secretRef:
          name: root-certificate-old
          namespace: gloo-mesh
  

Now that all your workloads point to the new root CA in the certificate chain, you can go ahead and remove the old root CA certificate. For setup instructions and a sample root trust policy, see the steps after the following diagram.

1. In your root trust policy, remove the passiveCertificateAuthorities section in your root trust policy. This step removes your old root CA certificate as a passive certificate that can be used to verify intermediate CA certificates.
2. Restart istiod to remove the old root CA certificate from the certificate chain.
3. To verify that workloads in your mesh can still communicate with each other, restart a non-critical workload or a test workload in your cluster so that it requests a new leaf certificate from istiod.
4. Send a request from your test workload to a workload in the service mesh that was not yet restarted and verify that these workloads can communicate with each other.
5. Optional: Restart all your workloads in the service mesh to remove the old root CA certificate from the certificate chain. Note that this step is optional as workloads renew their certificates within 24 hours by default.

Sample root trust policy:

  apiVersion: admin.gloo.solo.io/v2
kind: RootTrustPolicy
metadata:
  name: istio-ingressgateway
  namespace: gloo-mesh
spec:
  config:
    mgmtServerCa:
      secretRef:
        name: root-certificate-new
        namespace: gloo-mesh