Navigation :

Generating Istio certificates

Generate and provide Istio root CA and intermediate CA certificates in your Gloo Mesh setup so that each Istio deployment can issue certificates to workload pods in its mesh. You can choose from the following options:

Use built-in Istio tools to generate root CA and intermediate CA certificates
Manually manage your own root and intermediate CA Istio certificates in each workload cluster.
Bring your own root and intermediate CA certificates, but let Gloo Mesh manage signing the Istio certs in each workload cluster for you.

Before you begin

This guide assumes that you use the same names for components like clusters, workspaces, and namespaces as in the getting started. If you have different names, make sure to update the sample configuration files in this guide.

Complete the multicluster getting started guide to set up the following testing environment.
- Three clusters along with environment variables for the clusters and their Kubernetes contexts.
- The Gloo Platform CLI, meshctl, along with other CLI tools such as kubectl and istioctl.
- The Gloo management server in the management cluster, and the Gloo agents in the workload clusters.
- Istio installed in the workload clusters.
- A simple Gloo workspace setup.
Install Bookinfo and other sample apps.
The default openssl version that is included in macOS is LibreSSL, which does not work with these instructions.

Make sure that you have the OpenSSL version of openssl, not LibreSSL. The openssl version must be at least 1.1.
1. Check the openssl version that is installed. If you see LibreSSL in the output, continue to the next step.
```
openssl version
```
2. Install the OpenSSL version (not LibreSSL). For example, you might use Homebrew.
```
brew install openssl
```
3. Review the output of the OpenSSL installation for the path of the binary file. You can choose to export the binary to your path, or call the entire path whenever the following steps use an openssl command.
  - For example, openssl might be installed along the following path: /usr/local/opt/openssl@3/bin/
  - To run commands, you can append the path so that your terminal uses this installed version of OpenSSL, and not the default LibreSSL. /usr/local/opt/openssl@3/bin/openssl req -new -newkey rsa:4096 -x509 -sha256 -days 3650...
Save the kubeconfig contexts for your clusters. Run kubectl config get-contexts, look for your cluster in the CLUSTER column, and get the context name in the NAME column. Note: Do not use context names with underscores. The context name is used as a SAN specification in the generated certificate that connects workload clusters to the management cluster, and underscores in SAN are not FQDN compliant. You can rename a context by running kubectl config rename-context "<oldcontext>" <newcontext>.
```
export MGMT_CLUSTER=<mgmt-cluster-name>
export REMOTE_CLUSTER=<remote-cluster-name>
export MGMT_CONTEXT=<management-cluster-context>
export REMOTE_CONTEXT=<remote-cluster-context>
```

Using Istio tools to generate root CA and intermediate CA certificates

Istio provides a certificate tool that you can use to quickly generate the root CA and intermediate CA certificates for Istio deployments in each cluster.

Switch to the management cluster context.

kubectl config use-context $MGMT_CONTEXT

Download Istio in the management cluster. Make sure to use a supported version that matches the version you plan to run in the workload clusters.
```
ISTIO_VERSION=1.16.2
curl -L https://istio.io/downloadIstio | ISTIO_VERSION=$ISTIO_VERSION sh -
```

Set the variables for each CA that you can override when generating your certificates. For the full list of variables, see the following example root-ca.conf.

#------------------------------------------------------------------------
# variables: root CA
ROOTCA_DAYS ?= 3650
ROOTCA_KEYSZ ?= 4096
ROOTCA_ORG ?= Istio
ROOTCA_CN ?= Root CA
KUBECONFIG ?= $(HOME)/.kube/config
ISTIO_NAMESPACE ?= istio-system

#------------------------------------------------------------------------
# variables: intermediate CA
INTERMEDIATE_DAYS ?= 730
INTERMEDIATE_KEYSZ ?= 4096
INTERMEDIATE_ORG ?= Istio
INTERMEDIATE_CN ?= Intermediate CA
INTERMEDIATE_SAN_DNS ?= istiod.istio-system.svc

Generate the root certificate.

# Go to certs directory
cd istio-$ISTIO_VERSION/tools/certs

# Create root certificate
make -f Makefile.selfsigned.mk \
  ROOTCA_CN="Solo Root CA" \
  ROOTCA_ORG=solo.io \
  root-ca


# If needed, delete certs
# make -f Makefile.selfsigned.mk clean

Generate the intermediate CA certificate for the relay agent in a workload cluster. Repeat this step for each workload cluster you plan to register with Gloo Mesh.

REMOTE_CLUSTER1=<remote-cluster-1-name>
REMOTE_CONTEXT1=<remote-cluster-1-context>

make -f Makefile.selfsigned.mk \
  INTERMEDIATE_CN="Solo Intermediate CA" \
  INTERMEDIATE_ORG=solo.io \
  $REMOTE_CLUSTER1-cacerts

# Apply Kubernetes secret to cluster
kubectl create secret generic cacerts --context $REMOTE_CONTEXT1 -n istio-system \
      --from-file=$REMOTE_CLUSTER1/ca-cert.pem \
      --from-file=$REMOTE_CLUSTER1/ca-key.pem \
      --from-file=$REMOTE_CLUSTER1/root-cert.pem \
      --from-file=$REMOTE_CLUSTER1/cert-chain.pem

Create your own root and intermediate CAs to generate certificates

By default, Gloo Mesh can generate and self-sign certificates with the built-in root and intermediate signing CAs ) that Istio uses to establish mutual TLS connectivity across your services. In this setup, the root and intermediate certificates are stored in the management cluster, which can be a security risk.

To build your own certificate chain of trust and control how the certificates are signed and managed, you can create the certificates for the root and intermediate CAs in your own PKI infrastructure. Then, you manually provide the certificates as Kubernetes secret in each workload cluster. This setup is similar to the Both external to cluster approach in the Certificate architectures page ).

In this setup, you are responsible for managing and rotating the certificates. Typically, you use a tool, such as AWS Certificate Manager (ACM), to automate these steps.

Create the root and intermediate CA configurations in the following format.

# Root CA example configuration
cat > "root-ca.conf" <<EOF
[ req ]
encrypt_key = no
prompt = no
utf8 = yes
default_md = sha256
default_bits = 4096
req_extensions = req_ext
x509_extensions = req_ext
distinguished_name = req_dn
[ req_ext ]
subjectKeyIdentifier = hash
basicConstraints = critical, CA:true
keyUsage = critical, digitalSignature, nonRepudiation, keyEncipherment, keyCertSign
[ req_dn ]
O = Istio
CN = Root CA
EOF
   
# Istio intermediate CA example configuration
cat > "cluster-ca.conf" <<EOF
[ req ]
encrypt_key = no
prompt = no
utf8 = yes
default_md = sha256
default_bits = 4096
req_extensions = req_ext
x509_extensions = req_ext
distinguished_name = req_dn
[ req_ext ]
subjectKeyIdentifier = hash
basicConstraints = critical, CA:true, pathlen:0
keyUsage = critical, digitalSignature, nonRepudiation, keyEncipherment, keyCertSign
subjectAltName=@san
[ san ]
DNS.1 = istiod.istio-system.svc
[ req_dn ]
O = Istio
CN = Intermediate CA
L = ${cluster}
EOF

Using the CA configurations, generate the root key and certificate, and the intermediate signing key, CSR, and certificate. The following example uses OpenSSL, but you might use your certificate manager instead.

#!/bin/bash
## Generate a root key and use this key to create a self-signed certificate for the root CA
openssl req -new -newkey -config root-ca.conf -nodes -out root-ca.crt -keyout root-ca.key

## Create an intermediate CA key and the certificate signing request for the intermediate CA 
## Use the root CA key and certificate to create a signed certificate for the intermediate CA
openssl genrsa -out istio-signing-ca.key 4096
openssl req -new -key istio-signing-ca.key -out istio-signing-ca.csr -config cluster-ca.conf

openssl x509 -req \
  -days 3650 \
  -CA root-ca.crt -CAkey root-ca.key \
  -set_serial 0 \
  -in istio-signing-ca.csr -out -istio-signing-ca.crt \
  -extensions req_ext -extfile cluster-ca.conf

## Create the certificate chain with the signed root and intermediate CA certificates
cat root-ca.crt > cert-chain.pem
cat istio-signing-ca.crt >> cert-chain.pem

Provide the certificates as a Kubernetes secret named cacerts in the istio-system namespace. The file names are important. For example, the root-cert.pem is the root certificate that you previously created at root-ca.crt.

kubectl create secret generic cacerts -n istio-system --context $REMOTE_CONTEXT \
  --from-file=root-cert.pem=root-ca.crt \
  --from-file=ca-cert.pem=istio-signing-ca.crt \
  --from-file=ca-key.pem=istio-signing-ca.key \
  --from-file=cert-chain.pem=cert-chain.pem

Repeat the previous step for each workload cluster in your setup.

Bring your own root and intermediate certificates

By default, Gloo Mesh can generate and self-sign certificates with the built-in root and intermediate CAs ) that Istio uses to establish mutual TLS connectivity across your services. In this setup, the root and intermediate certificates are stored in the management cluster, which can be a security risk.

To enhance the security of your setup, keep the root key outside the cluster and store it in your own public key infrastructure (PKI) provider. Then, provide your own intermediate certificate as a Kubernetes secret in the management cluster.

Gloo Mesh uses the intermediate certificate to manage the Istio certificates in each workload cluster. This setup is similar to the Hybrid approach in the Certificate architectures page ).

Create the root and intermediate keys and certificates outside your Kubernetes cluster. The following example uses OpenSSL, but you might use your own certificate manager instead.

#!/bin/bash

rm -rf data/certs
mkdir -p data/certs

## Generate root certificate
openssl req -new -newkey rsa:4096 -x509 -sha256 \
  -days 3650 -nodes -out data/certs/root-ca.crt -keyout data/certs/root-ca.key \
  -subj "/CN=root-ca" \
  -addext "extendedKeyUsage = clientAuth, serverAuth"

## Configure the Istio intermediate CA
cat > "data/certs/gloo-mesh-istio-signing-ca.conf" <<EOF
[ req ]
encrypt_key = no
prompt = no
utf8 = yes
default_md = sha256
default_bits = 4096
req_extensions = req_ext
x509_extensions = req_ext
distinguished_name = req_dn
[ req_ext ]
subjectKeyIdentifier = hash
basicConstraints = critical, CA:true
keyUsage = critical, digitalSignature, nonRepudiation, keyEncipherment, keyCertSign
[ req_dn ]
O = Istio
CN = Root CA
EOF

## Create an intermediate key and certificate signing request (CSR)
## Use the root key and certificate to create a signed certificate for the intermediate CA
openssl genrsa -out data/certs/gloo-mesh-istio-signing-ca.key 4096
openssl req -new -key data/certs/gloo-mesh-istio-signing-ca.key -out data/certs/gloo-mesh-istio-signing-ca.csr -config data/certs/gloo-mesh-istio-signing-ca.conf

openssl x509 -req \
  -days 3650 \
  -CA data/certs/root-ca.crt -CAkey data/certs/root-ca.key \
  -set_serial 0 \
  -in data/certs/gloo-mesh-istio-signing-ca.csr -out data/certs/gloo-mesh-istio-signing-ca.crt \
  -extensions req_ext -extfile data/certs/gloo-mesh-istio-signing-ca.conf

## Create the certificate chain with the root and intermediate certificates
cat data/certs/root-ca.crt > data/certs/cert-chain.pem
cat data/certs/gloo-mesh-istio-signing-ca.crt >> data/certs/cert-chain.pem

Provide the root certificate, and the Istio intermediate CA certificate and key to the Gloo Mesh management plane as a Kubernetes secret. The file names are important. For example, the root-cert.pem is the root certificate that you previously created at data/certs/root-ca.crt.

kubectl create secret generic gloo-mesh-istio-signing-ca -n gloo-mesh --context $MGMT_CONTEXT \
  --from-file=root-cert.pem=data/certs/root-ca.crt \
  --from-file=ca-cert.pem=data/certs/gloo-mesh-istio-signing-ca.crt \
  --from-file=ca-key.pem=data/certs/gloo-mesh-istio-signing-ca.key \
  --from-file=cert-chain.pem=data/certs/cert-chain.pem

Create a root trust policy that refers to the Kubernetes secret. For more information, see the API docs. Gloo Mesh uses this intermediate CA to sign the Istio certificates for each workload cluster that is added to your mesh.

This example includes the autoRestartPods: true setting. Gloo Mesh will restart all of the Istio workloads in all of the cluster, to speed up the certificate rotation for the workloads. To avoid downtime, do NOT set this field to true in production environments. Instead, see Rotating certificates for Istio workloads.
```
apiVersion: admin.gloo.solo.io/v2
kind: RootTrustPolicy
metadata:
  name: root-trust-policy
  namespace: gloo-mesh
spec:
  config:
    mgmtServerCa:
      secretRef:
        name: gloo-mesh-istio-signing-ca
        namespace: gloo-mesh
    autoRestartPods: true
```

Rotating certificates for Istio workloads

When certificates are issued, pods that are managed by Istio must be restarted to ensure they pick up the new certificates. The certificate issuer creates a PodBounceDirective, which contains the namespaces and labels of the pods that must be restarted. For more information about how certificate rotation works in Istio, review the video series in this blog post.

Note: To avoid potential downtime for your apps in production, disable the PodBounceDirective feature by setting autoRestartPods to false. Then, control pod restarts in another way, such as a rolling update.

Get your root trust policies.

kubectl get roottrustpolicy --context ${MGMT_CONTEXT} -A

In the root trust policy, remove or set the autoRestartPods field to false.

kubectl edit roottrustpolicy --context ${MGMT_CONTEXT} -n <namespace> <root-trust-policy>

apiVersion: admin.gloo.solo.io/v2
kind: RootTrustPolicy
metadata:
  name: istio-ingressgateway
  namespace: gloo-mesh
spec:
  config:
    autoRestartPods: false
    ...

To ensure pods pick up the new certificates, restart the istiod pod in each remote cluster.

kubectl --context {$REMOTE_CONTEXT} -n istio-system patch deployment istiod \
    -p "{\"spec\":{\"template\":{\"metadata\":{\"labels\":{\"date\":\"`date +'%s'`\"}}}}}"

Restart your app pods that are managed by Istio, such as by using a rolling update strategy.