Generating Istio certificates

Generate and provide Istio root CA and intermediate CA certificates in your Gloo Mesh setup so that each Istio deployment can issue certificates to workload pods in its mesh. You can choose from the following options:

Using Istio tools to generate root CA and intermediate CA certificates

Istio provides a certificate tool that you can use to quickly generate the root CA and intermediate CA certificates for Istio deployments in each cluster.

  1. Switch to the management cluster context.

    kubectl config use-context $MGMT_CONTEXT
    
  2. Download Istio in the management cluster. Make sure to use a supported version that matches the version you plan to run in the workload clusters.

    ISTIO_VERSION=1.13.4
    curl -L https://istio.io/downloadIstio | ISTIO_VERSION=$ISTIO_VERSION sh -
    
  3. Set the variables for each CA that you can override when generating your certificates. For the full list of variables, see the following example root-ca.conf.

    #------------------------------------------------------------------------
    # variables: root CA
    ROOTCA_DAYS ?= 3650
    ROOTCA_KEYSZ ?= 4096
    ROOTCA_ORG ?= Istio
    ROOTCA_CN ?= Root CA
    KUBECONFIG ?= $(HOME)/.kube/config
    ISTIO_NAMESPACE ?= istio-system
    
    #------------------------------------------------------------------------
    # variables: intermediate CA
    INTERMEDIATE_DAYS ?= 730
    INTERMEDIATE_KEYSZ ?= 4096
    INTERMEDIATE_ORG ?= Istio
    INTERMEDIATE_CN ?= Intermediate CA
    INTERMEDIATE_SAN_DNS ?= istiod.istio-system.svc
    
  4. Generate the root certificate.

    # Go to certs directory
    cd istio-$ISTIO_VERSION/tools/certs
    
    # Create root certificate
    make -f Makefile.selfsigned.mk \
      ROOTCA_CN="Solo Root CA" \
      ROOTCA_ORG=solo.io \
      root-ca
    
    
    # If needed, delete certs
    # make -f Makefile.selfsigned.mk clean
    
  5. Generate the intermediate CA certificate for the relay agent in a workload cluster. Repeat this step for each workload cluster you plan to register with Gloo Mesh.

    REMOTE_CLUSTER1=<remote-cluster-1-name>
    REMOTE_CONTEXT1=<remote-cluster-1-context>
    
    make -f Makefile.selfsigned.mk \
      INTERMEDIATE_CN="Solo Intermediate CA" \
      INTERMEDIATE_ORG=solo.io \
      $REMOTE_CLUSTER1-cacerts
    
    # Apply Kubernetes secret to cluster
    kubectl create secret generic cacerts --context $REMOTE_CONTEXT1 -n istio-system \
          --from-file=$REMOTE_CLUSTER1/ca-cert.pem \
          --from-file=$REMOTE_CLUSTER1/ca-key.pem \
          --from-file=$REMOTE_CLUSTER1/root-cert.pem \
          --from-file=$REMOTE_CLUSTER1/cert-chain.pem
    

Creating root and intermediate CAs to generate certificates

To create CAs in your own PKI infrastructure, you can use these example configurations as a starting point for a root and intermediate CAs that can generate Istio intermediate CA certificates with the same root for each workload cluster.

# Root CA example configuration
cat > "root-ca.conf" <<EOF
[ req ]
encrypt_key = no
prompt = no
utf8 = yes
default_md = sha256
default_bits = 4096
req_extensions = req_ext
x509_extensions = req_ext
distinguished_name = req_dn
[ req_ext ]
subjectKeyIdentifier = hash
basicConstraints = critical, CA:true
keyUsage = critical, digitalSignature, nonRepudiation, keyEncipherment, keyCertSign
[ req_dn ]
O = Istio
CN = Root CA
EOF

# Istio intermediate signing CA example configuration
cat > "cluster-ca.conf" <<EOF
[ req ]
encrypt_key = no
prompt = no
utf8 = yes
default_md = sha256
default_bits = 4096
req_extensions = req_ext
x509_extensions = req_ext
distinguished_name = req_dn
[ req_ext ]
subjectKeyIdentifier = hash
basicConstraints = critical, CA:true, pathlen:0
keyUsage = critical, digitalSignature, nonRepudiation, keyEncipherment, keyCertSign
subjectAltName=@san
[ san ]
DNS.1 = istiod.istio-system.svc
[ req_dn ]
O = Istio
CN = Intermediate CA
L = ${cluster}
EOF

Manually providing Istio root CA and intermediate CA certificates

You can bring your own signing certificates and store them as Kubernetes secrets within each workload cluster. Note that adding automation to create and rotate these certificates, such as using AWS Certificate Manager (ACM), is recommended.

The name of the secret must be cacerts, and it must be created in the istio-system namespace of each workload cluster. The files in the secret must be named in the same format as the following.

# Example kubectl apply (file names are important)
# ca-cert.pem
# ca-key.pem
# root-cert.pem
# cert-chain.pem
kubectl create secret generic cacerts -n istio-system \
      --from-file=ca-cert.pem=istio-ca/cert.pem \
      --from-file=ca-key.pem=istio-ca/ca.key.decrypted \
      --from-file=root-cert.pem=root-ca/ca.crt \
      --from-file=cert-chain.pem=istio-ca/cert-chain.pem

Rotating certificates for Istio workloads

When certificates are issued, pods that are managed by Istio must be restarted to ensure they pick up the new certificates. The certificate issuer creates a PodBounceDirective, which contains the namespaces and labels of the pods that must be restarted. For more information about how certificate rotation works in Istio, review the video series in this blog post.

Note: To avoid potential downtime for your apps in production, disable the PodBounceDirective feature. Then, control pod restarts in another way, such as a rolling update.

  1. Get your root trust policies.

    kubectl get roottrustpolicy --context ${MGMT_CONTEXT} -A
    
  2. In the root trust policy, set the autoRestartPods field to false.

    kubectl edit roottrustpolicy --context ${MGMT_CONTEXT} -n NAMESPACE ROOT_TRUST_POLICY
    
    apiVersion: admin.gloo.solo.io/v2
    kind: RootTrustPolicy
    metadata:
      name: north-south-gw
      namespace: gloo-mesh
    spec:
      config:
        autoRestartPods: false
        ...
    
  3. To ensure pods pick up the new certificates, restart the istiod pod in each remote cluster.

    kubectl --context {$REMOTE_CONTEXT} -n istio-system patch deployment istiod \
        -p "{\"spec\":{\"template\":{\"metadata\":{\"labels\":{\"date\":\"`date +'%s'`\"}}}}}"
    
  4. Restart your app pods that are managed by Istio, such as by using a rolling update strategy.