Generating Istio certificates

Generate and provide Istio root CA and intermediate CA certificates in your Gloo Mesh setup so that each Istio deployment can issue certificates to workload pods in its mesh. You can choose from the following options:

Before you begin

This guide assumes that you use the same names for components like clusters, workspaces, and namespaces as in the getting started. If you have different names, make sure to update the sample configuration files in this guide.
  1. Complete the multicluster getting started guide to set up the following testing environment.

    • Three clusters along with environment variables for the clusters and their Kubernetes contexts.
    • The Gloo Platform CLI, meshctl, along with other CLI tools such as kubectl and istioctl.
    • The Gloo management server in the management cluster, and the Gloo agents in the workload clusters.
    • Istio installed in the workload clusters.
    • A simple Gloo workspace setup.
  2. Install Bookinfo and other sample apps.

  3. The default openssl version that is included in macOS is LibreSSL, which does not work with these instructions.

    Make sure that you have the OpenSSL version of openssl, not LibreSSL. The openssl version must be at least 1.1.

    1. Check the openssl version that is installed. If you see LibreSSL in the output, continue to the next step.
      openssl version
      
    2. Install the OpenSSL version (not LibreSSL). For example, you might use Homebrew.
      brew install openssl
      
    3. Review the output of the OpenSSL installation for the path of the binary file. You can choose to export the binary to your path, or call the entire path whenever the following steps use an openssl command.
      • For example, openssl might be installed along the following path: /usr/local/opt/openssl@3/bin/
      • To run commands, you can append the path so that your terminal uses this installed version of OpenSSL, and not the default LibreSSL. /usr/local/opt/openssl@3/bin/openssl req -new -newkey rsa:4096 -x509 -sha256 -days 3650...

  4. Save the kubeconfig contexts for your clusters. Run kubectl config get-contexts, look for your cluster in the CLUSTER column, and get the context name in the NAME column. Note: Do not use context names with underscores. The context name is used as a SAN specification in the generated certificate that connects workload clusters to the management cluster, and underscores in SAN are not FQDN compliant. You can rename a context by running kubectl config rename-context "<oldcontext>" <newcontext>.
    export MGMT_CLUSTER=<mgmt-cluster-name>
    export REMOTE_CLUSTER=<remote-cluster-name>
    export MGMT_CONTEXT=<management-cluster-context>
    export REMOTE_CONTEXT=<remote-cluster-context>
    

Using Istio tools to generate root CA and intermediate CA certificates

Istio provides a certificate tool that you can use to quickly generate the root CA and intermediate CA certificates for Istio deployments in each cluster.

  1. Switch to the management cluster context.

    kubectl config use-context $MGMT_CONTEXT
    
  2. Download Istio in the management cluster. Make sure to use a supported version that matches the version you plan to run in the workload clusters.

    ISTIO_VERSION=1.16.2
    curl -L https://istio.io/downloadIstio | ISTIO_VERSION=$ISTIO_VERSION sh -
    
  3. Set the variables for each CA that you can override when generating your certificates. For the full list of variables, see the following example root-ca.conf.

    #------------------------------------------------------------------------
    # variables: root CA
    ROOTCA_DAYS ?= 3650
    ROOTCA_KEYSZ ?= 4096
    ROOTCA_ORG ?= Istio
    ROOTCA_CN ?= Root CA
    KUBECONFIG ?= $(HOME)/.kube/config
    ISTIO_NAMESPACE ?= istio-system
    
    #------------------------------------------------------------------------
    # variables: intermediate CA
    INTERMEDIATE_DAYS ?= 730
    INTERMEDIATE_KEYSZ ?= 4096
    INTERMEDIATE_ORG ?= Istio
    INTERMEDIATE_CN ?= Intermediate CA
    INTERMEDIATE_SAN_DNS ?= istiod.istio-system.svc
    
  4. Generate the root certificate.

    # Go to certs directory
    cd istio-$ISTIO_VERSION/tools/certs
    
    # Create root certificate
    make -f Makefile.selfsigned.mk \
      ROOTCA_CN="Solo Root CA" \
      ROOTCA_ORG=solo.io \
      root-ca
    
    
    # If needed, delete certs
    # make -f Makefile.selfsigned.mk clean
    
  5. Generate the intermediate CA certificate for the relay agent in a workload cluster. Repeat this step for each workload cluster you plan to register with Gloo Mesh.

    REMOTE_CLUSTER1=<remote-cluster-1-name>
    REMOTE_CONTEXT1=<remote-cluster-1-context>
    
    make -f Makefile.selfsigned.mk \
      INTERMEDIATE_CN="Solo Intermediate CA" \
      INTERMEDIATE_ORG=solo.io \
      $REMOTE_CLUSTER1-cacerts
    
    # Apply Kubernetes secret to cluster
    kubectl create secret generic cacerts --context $REMOTE_CONTEXT1 -n istio-system \
          --from-file=$REMOTE_CLUSTER1/ca-cert.pem \
          --from-file=$REMOTE_CLUSTER1/ca-key.pem \
          --from-file=$REMOTE_CLUSTER1/root-cert.pem \
          --from-file=$REMOTE_CLUSTER1/cert-chain.pem
    

Create your own root and intermediate CAs to generate certificates

By default, Gloo Mesh can generate and self-sign certificates with the built-in root and intermediate signing CAs ) that Istio uses to establish mutual TLS connectivity across your services. In this setup, the root and intermediate certificates are stored in the management cluster, which can be a security risk.

To build your own certificate chain of trust and control how the certificates are signed and managed, you can create the certificates for the root and intermediate CAs in your own PKI infrastructure. Then, you manually provide the certificates as Kubernetes secret in each workload cluster. This setup is similar to the Both external to cluster approach in the Certificate architectures page ).

In this setup, you are responsible for managing and rotating the certificates. Typically, you use a tool, such as AWS Certificate Manager (ACM), to automate these steps.

  1. Create the root and intermediate CA configurations in the following format.

    # Root CA example configuration
    cat > "root-ca.conf" <<EOF
    [ req ]
    encrypt_key = no
    prompt = no
    utf8 = yes
    default_md = sha256
    default_bits = 4096
    req_extensions = req_ext
    x509_extensions = req_ext
    distinguished_name = req_dn
    [ req_ext ]
    subjectKeyIdentifier = hash
    basicConstraints = critical, CA:true
    keyUsage = critical, digitalSignature, nonRepudiation, keyEncipherment, keyCertSign
    [ req_dn ]
    O = Istio
    CN = Root CA
    EOF
       
    # Istio intermediate CA example configuration
    cat > "cluster-ca.conf" <<EOF
    [ req ]
    encrypt_key = no
    prompt = no
    utf8 = yes
    default_md = sha256
    default_bits = 4096
    req_extensions = req_ext
    x509_extensions = req_ext
    distinguished_name = req_dn
    [ req_ext ]
    subjectKeyIdentifier = hash
    basicConstraints = critical, CA:true, pathlen:0
    keyUsage = critical, digitalSignature, nonRepudiation, keyEncipherment, keyCertSign
    subjectAltName=@san
    [ san ]
    DNS.1 = istiod.istio-system.svc
    [ req_dn ]
    O = Istio
    CN = Intermediate CA
    L = ${cluster}
    EOF
    
  2. Using the CA configurations, generate the root key and certificate, and the intermediate signing key, CSR, and certificate. The following example uses OpenSSL, but you might use your certificate manager instead.

    #!/bin/bash
    ## Generate a root key and use this key to create a self-signed certificate for the root CA
    openssl req -new -newkey -config root-ca.conf -nodes -out root-ca.crt -keyout root-ca.key
    
    ## Create an intermediate CA key and the certificate signing request for the intermediate CA 
    ## Use the root CA key and certificate to create a signed certificate for the intermediate CA
    openssl genrsa -out istio-signing-ca.key 4096
    openssl req -new -key istio-signing-ca.key -out istio-signing-ca.csr -config cluster-ca.conf
    ​
    openssl x509 -req \
      -days 3650 \
      -CA root-ca.crt -CAkey root-ca.key \
      -set_serial 0 \
      -in istio-signing-ca.csr -out -istio-signing-ca.crt \
      -extensions req_ext -extfile cluster-ca.conf
    ​​
    ## Create the certificate chain with the signed root and intermediate CA certificates
    cat root-ca.crt > cert-chain.pem
    cat istio-signing-ca.crt >> cert-chain.pem
       ​

  3. Provide the certificates as a Kubernetes secret named cacerts in the istio-system namespace. The file names are important. For example, the root-cert.pem is the root certificate that you previously created at root-ca.crt.

    kubectl create secret generic cacerts -n istio-system --context $REMOTE_CONTEXT \
      --from-file=root-cert.pem=root-ca.crt \
      --from-file=ca-cert.pem=istio-signing-ca.crt \
      --from-file=ca-key.pem=istio-signing-ca.key \
      --from-file=cert-chain.pem=cert-chain.pem
       
  4. Repeat the previous step for each workload cluster in your setup.

Bring your own root and intermediate certificates

By default, Gloo Mesh can generate and self-sign certificates with the built-in root and intermediate CAs ) that Istio uses to establish mutual TLS connectivity across your services. In this setup, the root and intermediate certificates are stored in the management cluster, which can be a security risk.

To enhance the security of your setup, keep the root key outside the cluster and store it in your own public key infrastructure (PKI) provider. Then, provide your own intermediate certificate as a Kubernetes secret in the management cluster.

Gloo Mesh uses the intermediate certificate to manage the Istio certificates in each workload cluster. This setup is similar to the Hybrid approach in the Certificate architectures page ).

  1. Create the root and intermediate keys and certificates outside your Kubernetes cluster. The following example uses OpenSSL, but you might use your own certificate manager instead.

    #!/bin/bash
    ​
    rm -rf data/certs
    mkdir -p data/certs
    
    ## Generate root certificate​
    openssl req -new -newkey rsa:4096 -x509 -sha256 \
      -days 3650 -nodes -out data/certs/root-ca.crt -keyout data/certs/root-ca.key \
      -subj "/CN=root-ca" \
      -addext "extendedKeyUsage = clientAuth, serverAuth"## Configure the Istio intermediate CA
    cat > "data/certs/gloo-mesh-istio-signing-ca.conf" <<EOF
    [ req ]
    encrypt_key = no
    prompt = no
    utf8 = yes
    default_md = sha256
    default_bits = 4096
    req_extensions = req_ext
    x509_extensions = req_ext
    distinguished_name = req_dn
    [ req_ext ]
    subjectKeyIdentifier = hash
    basicConstraints = critical, CA:true
    keyUsage = critical, digitalSignature, nonRepudiation, keyEncipherment, keyCertSign
    [ req_dn ]
    O = Istio
    CN = Root CA
    EOF
    
    ## Create an intermediate key and certificate signing request (CSR)
    ## Use the root key and certificate to create a signed certificate for the intermediate CA
    openssl genrsa -out data/certs/gloo-mesh-istio-signing-ca.key 4096
    openssl req -new -key data/certs/gloo-mesh-istio-signing-ca.key -out data/certs/gloo-mesh-istio-signing-ca.csr -config data/certs/gloo-mesh-istio-signing-ca.conf
    ​
    openssl x509 -req \
      -days 3650 \
      -CA data/certs/root-ca.crt -CAkey data/certs/root-ca.key \
      -set_serial 0 \
      -in data/certs/gloo-mesh-istio-signing-ca.csr -out data/certs/gloo-mesh-istio-signing-ca.crt \
      -extensions req_ext -extfile data/certs/gloo-mesh-istio-signing-ca.conf
    ​​
    ## Create the certificate chain with the root and intermediate certificates
    cat data/certs/root-ca.crt > data/certs/cert-chain.pem
    cat data/certs/gloo-mesh-istio-signing-ca.crt >> data/certs/cert-chain.pem
       ​
  2. Provide the root certificate, and the Istio intermediate CA certificate and key to the Gloo Mesh management plane as a Kubernetes secret. The file names are important. For example, the root-cert.pem is the root certificate that you previously created at data/certs/root-ca.crt.

    kubectl create secret generic gloo-mesh-istio-signing-ca -n gloo-mesh --context $MGMT_CONTEXT \
      --from-file=root-cert.pem=data/certs/root-ca.crt \
      --from-file=ca-cert.pem=data/certs/gloo-mesh-istio-signing-ca.crt \
      --from-file=ca-key.pem=data/certs/gloo-mesh-istio-signing-ca.key \
      --from-file=cert-chain.pem=data/certs/cert-chain.pem
       
  3. Create a root trust policy that refers to the Kubernetes secret. For more information, see the API docs. Gloo Mesh uses this intermediate CA to sign the Istio certificates for each workload cluster that is added to your mesh.

    This example includes the autoRestartPods: true setting. Gloo Mesh will restart all of the Istio workloads in all of the cluster, to speed up the certificate rotation for the workloads. To avoid downtime, do NOT set this field to true in production environments. Instead, see Rotating certificates for Istio workloads.

    apiVersion: admin.gloo.solo.io/v2
    kind: RootTrustPolicy
    metadata:
      name: root-trust-policy
      namespace: gloo-mesh
    spec:
      config:
        mgmtServerCa:
          secretRef:
            name: gloo-mesh-istio-signing-ca
            namespace: gloo-mesh
        autoRestartPods: true
    

Rotating certificates for Istio workloads

When certificates are issued, pods that are managed by Istio must be restarted to ensure they pick up the new certificates. The certificate issuer creates a PodBounceDirective, which contains the namespaces and labels of the pods that must be restarted. For more information about how certificate rotation works in Istio, review the video series in this blog post.

Note: To avoid potential downtime for your apps in production, disable the PodBounceDirective feature by setting autoRestartPods to false. Then, control pod restarts in another way, such as a rolling update.

  1. Get your root trust policies.

    kubectl get roottrustpolicy --context ${MGMT_CONTEXT} -A
    
  2. In the root trust policy, remove or set the autoRestartPods field to false.

    kubectl edit roottrustpolicy --context ${MGMT_CONTEXT} -n <namespace> <root-trust-policy>
    
    apiVersion: admin.gloo.solo.io/v2
    kind: RootTrustPolicy
    metadata:
      name: istio-ingressgateway
      namespace: gloo-mesh
    spec:
      config:
        autoRestartPods: false
        ...
    
  3. To ensure pods pick up the new certificates, restart the istiod pod in each remote cluster.

    kubectl --context {$REMOTE_CONTEXT} -n istio-system patch deployment istiod \
        -p "{\"spec\":{\"template\":{\"metadata\":{\"labels\":{\"date\":\"`date +'%s'`\"}}}}}"
    
  4. Restart your app pods that are managed by Istio, such as by using a rolling update strategy.